NVIDIA Dynamo Guide: Scaling Distributed LLM Inference on Bare Metal

1. NVIDIA Dynamo Architecture: Request, Event, and Transfer Planes

A common technical fallacy is treating Dynamo as a direct replacement for inference engines. Dynamo is an orchestration framework that sits above these engines (like vLLM or SGLang), transforming disparate GPU workers into a single, coordinated inference system.

To understand how Dynamo coordinates these workers, it helps to conceptualize the architecture across three distinct communication paths, alongside an underlying Service Discovery mechanism:

• Request Plane (RPC): Handles incoming traffic between the Frontend and workers (typically utilizing TCP or NATS).
• Event Plane (Telemetry & Signals): Asynchronously broadcasts KV events and worker telemetry (utilizing ZMQ or NATS) to keep the router updated on cache availability.
• KV Transfer Path (NIXL): Provides high-throughput data movement, including KV-cache transfers between Prefill and Decode workers in disaggregated serving. NIXL itself is a broader transfer abstraction supporting multiple memory and storage backends.

2. Aggregated vs. Disaggregated Serving Trade-Offs

LLM Inference consists of two distinct phases: Prefill (reading the prompt, which is typically compute-intensive) and Decode (generating tokens, which is typically memory-bandwidth-intensive). Dynamo offers two primary serving topologies:

• Aggregated Serving: Prefill and decode run on the same worker pool, minimizing network dependency and operating highly efficiently for shorter requests.
• Disaggregated Serving: Separates prefill and decode tasks onto distinct GPU pools tailored to the respective bottlenecks.

3. NIXL Transport Backends and Topology-Aware KV Cache Transfer

In disaggregated serving, KV cache must move from Prefill to Decode workers via NIXL (NVIDIA Inference Xfer Library). NIXL dynamically selects a suitable backend based on the source/destination memory types and the backends initialized on both agents. Supported backends include:

• UCX: Supports InfiniBand, RoCE, and TCP for inter-node VRAM and DRAM transfers.
• Libfabric: Provides communication through supported OFI providers.
• GDS / GDS-MT: Enables GPU-to-storage transfers through GPUDirect Storage.
• POSIX: Supports conventional DRAM-to-file transfers.
• DOCA GPUNetIO, UCCL-P2P, Mooncake: Alternative high-performance transfer paths.

(Note: NVLink and NVSwitch provide the underlying high-bandwidth hardware fabrics; they should not be confused with NIXL's software backend names).

Topology-Aware KV Transfer

In multi-node deployments, selecting a decode worker based solely on capacity can introduce severe network latency. By publishing topology metadata for domains such as host, rack, or availability zone, operators can constrain or bias decode-worker selection toward a preferred KV-transfer domain.

In Kubernetes deployments, configure topology-aware transfer through the DGD's spec.experimental.kvTransferPolicy. Important constraints apply:

• Define either labelKey or clusterTopologyName, not both.
• Specify the corresponding domain (e.g., rack or zone).
• If using preferred enforcement, preferredWeight is required (it acts as a strength preference, not a probability).
• In required mode, requests will fail if no eligible decode worker exists in the selected domain.

While this reduces unnecessary network hops, actual GPU-NIC affinity and RDMA performance still depend on the physical topology, device configuration, and active NIXL transport.

4. KV-Aware Routing & The Evolution of Cache Management

Dynamo’s Smart Router performs Cache Locality + Load-Aware Placement. Through the Event Plane, workers actively publish KV block lifecycle events. The router evaluates prefix overlap and active worker load to route requests. (Note: If KV events are disabled or event delivery fails, the router falls back to predicting cache states based on past routing decisions).

5. Agentic Inference, Session Identity & Affinity

Agentic workflows often append tool outputs and reasoning context across multiple requests. When previous prompt prefixes cannot be reused effectively, repeated context processing increases prefill work and KV-cache pressure. Session-aware routing and prefix-aware cache reuse can help reduce unnecessary recomputation. Dynamo tracks these trajectories using headers like X-Dynamo-Session-ID, X-Dynamo-Session-Final, and X-Dynamo-Parent-Session-ID (useful for identifying session lineage in parent-child agent workflows).

Identity vs. Lifecycle and Session Affinity

Session affinity is a routing decision, not a cache-persistence guarantee. A session can remain pinned to a worker while its KV blocks are still subject to the backend engine's cache capacity and eviction policies. Passing a session ID does not automatically preserve KV cache indefinitely or enable sticky routing. You must configure explicit frontend routing policies (e.g., via --router-session-affinity-ttl-secs) in hard or soft binding modes.

The X-Dynamo-Session-Final header signals that an agent trajectory has ended. Lifecycle-aware consumers can use this signal to release associated session state early. However, KV block eviction remains subject to the configured cache-management lifecycle policies.

6. NVIDIA Dynamo Kubernetes Architecture and Orchestration Components

Deploying Dynamo is a multi-layered orchestration process on Kubernetes. It is crucial to understand the distinct responsibilities of each component in the stack:

• DGDR: Declares the deployment intent (Model, hardware constraints, SLA requirements).
• AIConfigurator / AISimulate: Provides hardware metadata, candidate configuration generation, performance estimation, and configuration ranking.
• Profiler: Evaluates candidate configurations to formulate the final deployment plan.
• Dynamo Operator: Reconciles the final DynamoGraphDeployment (DGD) lifecycle within Kubernetes.
• Grove Operator: Coordinates multi-node workload orchestration and gang scheduling.
• KAI Scheduler: Handles strict GPU-aware scheduling and resource allocation.

7. Deploy NVIDIA Dynamo v1.6.0 on Kubernetes

Below is the SRE-grade deployment path using the pinned v1.6.0 Dynamo platform and the v1beta1 API for an 8-GPU Single-Node example.

Step 1: Cluster Preparation & Prerequisites

Ensure your bare metal cluster has the NVIDIA GPU Operator installed. If your Kubernetes provider does not manage NVIDIA drivers natively, install it:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia --force-update
helm repo update nvidia

# Note: If NVIDIA drivers are already managed by your K8s provider, 
# append `--set driver.enabled=false` to avoid conflicts.
helm upgrade --install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --wait \
  --timeout=10m

Verify your GPU nodes have allocatable resources (ensure the node has 8 allocatable GPUs matching your target SKU):

kubectl get nodes -o custom-columns="NAME:.metadata.name,GPUs:.status.allocatable.nvidia\.com/gpu"

Step 2: Install the Dynamo Platform via Helm

To ensure version consistency and avoid tutorial drift, install the pinned platform chart directly.

export NAMESPACE="dynamo-system"
export DYNAMO_VERSION="1.6.0"

helm upgrade --install dynamo-platform \
  "https://helm.ngc.nvidia.com/nvidia/ai-dynamo/charts/dynamo-platform-${DYNAMO_VERSION}.tgz" \
  --namespace "$NAMESPACE" \
  --create-namespace \
  --wait \
  --timeout=10m

kubectl get pods --namespace "$NAMESPACE"

Step 3: Create Credentials (NGC & Hugging Face)

# 1. NGC Image Pull Secret
kubectl create secret docker-registry nvcr-imagepullsecret \
  --docker-server=nvcr.io \
  --docker-username='$oauthtoken' \
  --docker-password="" \
  -n "$NAMESPACE"

# 2. Hugging Face Token for Model Weights (Optional for public models)
kubectl create secret generic hf-token-secret \
  --namespace "$NAMESPACE" \
  --from-literal=HF_TOKEN="" \
  --dry-run=client -o yaml | kubectl apply -f -

Step 4: Deploy the Inference Graph (DGDR)

Submit a DynamoGraphDeploymentRequest (DGDR). Setting autoApply: false is highly recommended for production so you can manualy review the configuration before it goes live.

# Save this as dynamo-deploy.yaml
apiVersion: nvidia.com/v1beta1
kind: DynamoGraphDeploymentRequest
metadata:
  name: qwen2-5-inference
  namespace: dynamo-system
spec:
  model: Qwen/Qwen2.5-7B-Instruct
  backend: vllm
  
  image: nvcr.io/nvidia/ai-dynamo/dynamo-planner:1.6.0
  
  # 'rapid' uses AIC-backed simulation. Use 'thorough' for real GPU benchmarking.
  searchStrategy: rapid 
  
  # Illustrative values. Set realistic targets based on expected traffic.
  workload:
    isl: 3000
    osl: 150
    concurrency: 16
    
  hardware:
    gpuSku: h100_sxm
    totalGpus: 8
    numGpusPerNode: 8
    
  # Optimization targets for candidate selection (not guaranteed performance)
  sla:
    ttft: 200.0   
    itl: 20.0
    optimizationType: latency
    
  autoApply: false # Manual review required

kubectl apply -f dynamo-deploy.yaml

Step 5: Review the Generated DGD & Deploy

Watch the DGDR status. Once profiling reaches the Ready phase, extract, validate, and apply the generated configuration.

# Watch the status until it hits Ready phase
kubectl get dgdr qwen2-5-inference -n "$NAMESPACE" -w

# Extract the selected configuration
kubectl get dgdr qwen2-5-inference \
  -n "$NAMESPACE" \
  -o jsonpath='{.status.profilingResults.selectedConfig}' \
  > generated-dgd.yaml

# Inspect and validate the generated resource against the cluster API
cat generated-dgd.yaml
kubectl apply --dry-run=server -f generated-dgd.yaml -n "$NAMESPACE"

# Apply after reviewing the generated configuration
kubectl apply -f generated-dgd.yaml -n "$NAMESPACE"

# Verify the inference service components are running
kubectl get dgd -n "$NAMESPACE"
kubectl get pods -n "$NAMESPACE" -o wide
kubectl get svc -n "$NAMESPACE"

Step 6: Verify Inference & Test the Endpoint

Dynamo dynamically generates the frontend service name based on the DGD. Retrieve it programmatically.

# Retrieve the frontend service name
DGD_NAME=$(kubectl get dgdr/qwen2-5-inference \
  --namespace "$NAMESPACE" \
  -o jsonpath='{.status.dgdName}')
FRONTEND_SERVICE="${DGD_NAME}-frontend"

# Start port-forward
kubectl port-forward \
  "service/${FRONTEND_SERVICE}" \
  8000:8000 \
  --namespace "$NAMESPACE" \
  >/tmp/dynamo-port-forward.log 2>&1 &
PORT_FORWARD_PID=$!

# Wait for the frontend health endpoint
for i in $(seq 1 30); do
  if curl --silent --fail \
    http://localhost:8000/health >/dev/null; then
    echo "Dynamo frontend is ready."
    break
  fi
  sleep 2
done

# Send a test chat completion request
curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-7B-Instruct",
    "messages": [{"role": "user", "content": "Explain Kubernetes gang scheduling."}],
    "max_tokens": 150
  }'

Extending the Deployment to Multiple Nodes

The example above is scoped for a Single-Node 8-GPU deployment. For multi-node distributed inference, you must ensure your Kubernetes cluster is configured for gang scheduling. This typically requires prerequisites like the Grove Operator paired with the KAI Scheduler (or supported alternatives like LWS + Volcano). Without these orchestration prerequisites, multi-node workloads may deadlock during scheduling.

8. Operational Blueprint: Benchmarking, Planning & Recovery

An architecture that looks flawless on paper can easily bottleneck under live traffic without proper sizing.

A. Benchmarking Before Production

Evaluate your configuration using the thorough search strategy or real traffic generators:

• Aggregated vs. disaggregated workload comparisons under different concurrency levels.
• TTFT (Time to First Token) and ITL (Inter-Token Latency) strictly at P50, P95, and P99 percentiles. Averages hide tail latency.
• KV transfer latency and NIXL bandwidth under peak load.
• Tokens/sec/GPU and total cluster throughput.

B. Capacity Planning

Ensure your cluster is sized correctly by calculating:

• GPU memory required for both model weights and the expected KV cache (concurrent sequences).
• The optimal Prefill-to-Decode GPU ratio based on input/output token distribution.
• Network bandwidth required for NIXL KV transfers to prevent network congestion.
• Headroom for traffic spikes and worker failures.

C. Failure Recovery Scenarios

Test how the system behaves under stress:

• A prefill or decode worker crashes (Measure worker rescheduling and GPU resource recovery delays).
• NIXL transfer fails (Simulate RDMA link degradation).
• Restart the router and verify whether session-affinity bindings are preserved, reconstructed, or lost under the selected deployment configuration.

9. NVIDIA Dynamo Observability and SRE Troubleshooting

Use Prometheus and Grafana as the central observability stack. Collect Dynamo application metrics from its exposed metrics endpoints, NIXL transfer telemetry from the relevant runtime instrumentation, and GPU hardware metrics through NVIDIA DCGM Exporter. Correlate these signals (using request IDs) with Kubernetes pod events, node metrics, and network telemetry.

• Frontend & Router: Request rate, request failures, TTFT, routing overhead, and cache-aware routing efficiency.
• Prefill & Decode: Prompt processing latency, ITL, output tokens/sec, and active sequences.
• KV Cache & NIXL: Cache utilization, eviction pressure, prefix-cache hit behavior, and transfer duration.
• Kubernetes & Deployment Health: Pod restart count, worker readiness, Profiling Job status/duration, GPU scheduling failures, pending pods, DGD desired vs. available replicas, and Node pressure events.

10. Bare Metal vs. Public Cloud: The Infrastructure Reality

Bare Metal Dedicated Servers can provide greater control over GPU placement, PCIe topology, NUMA locality, NIC selection, and low-level network configuration, depending on the provider's hardware and provisioning model. Dedicated physical resources can reduce contention from neighboring tenants on allocated compute and memory resources. However, shared switches, uplinks, storage systems, and upstream network paths can still introduce contention.

Public cloud platforms can also offer RDMA-enabled networking, so the practical difference depends on the actual hardware topology, isolation model, and level of infrastructure control you maintain over the environment.

Conclusion

NVIDIA Dynamo is a paradigm shift in LLM serving, bridging the gap between raw inference engines and datacenter-scale orchestration. By mastering the Request, Event, and Transfer paths, utilizing topology-aware NIXL transfers, and relying on Kubernetes-native profiling, infrastructure teams can achieve unprecedented scaling.

Success relies on acknowledging that disaggregated serving is not a magic bullet. It demands rigorous SRE practices: configuring strict session affinity, transitioning away from deprecated KVBM patterns, enforcing deep P99 observability, and deploying on highly optimized physical infrastructure.