TensorRT-LLM Production Deployment: FP8, NVFP4, MoE & Disaggregated Serving

1. TensorRT-LLM Production Architecture & Backend Selection

Historically, TensorRT-LLM required a rigid, ahead-of-time (AOT) engine compilation phase (trtllm-build). As of the v1.2+ release line, TensorRT-LLM has transitioned toward a PyTorch-native runtime as its primary execution backend for standard serving, easing the legacy configuration wall while retaining optimized CUDA kernels.

Note on Pre-Releases: Configurations in this guide reference the 1.4.0rc2 release candidate. Features marked as prototypes or release candidates (like specific NVFP4 KV cache support) should be validated in a staging environment before production rollout, as their behavior may change in final stable tags.

2. FP8 and NVFP4 Quantization

Running a massive MoE model at FP16 is often computationally unviable for high-volume endpoints. TensorRT-LLM leverages the TensorRT Model Optimizer (ModelOpt) for Post-Training Quantization (PTQ).

• Hopper (H100/H200): Utilizes the FP8 E4M3 format. While theoretical throughput gains are frequently cited, actual performance depends heavily on sequence length, batch size, and the specific workload.
• Blackwell (B200/GB200): Unlocks the NVFP4 instruction path. FP4 reduces weight memory requirements theoretically by 4x compared to FP16, allowing large MoE models to fit within smaller NVLink domains.

Always validate quantization impact against a BF16 baseline using golden-prompt accuracy gates before production deployment.

3. TP, PP, and EP for MoE Models

Scaling MoE models requires precise Grouped-GEMM mappings across the interconnect fabric.

• Tensor Parallelism (TP): Distributes operations within layers across devices.
• Pipeline Parallelism (PP): Distributes model layers across stages.
• Expert Parallelism (EP): Shards distinct subsets of MoE experts across separate GPUs. Tokens are dynamically dispatched to their assigned experts and gathered back.

Topology-Dependent Sizing: There is no universally optimal EP size. Select EP, TP, and PP based on the model's architecture, GPU topology, supported parallelism configurations, and measured communication overhead. An eight-GPU NVSwitch node may be a useful starting point for intra-node experiments, while rack-scale systems (like the GB200 NVL72) enable larger distributed configurations. Benchmark alternative configurations using your target model and production request distribution.

4. Prefill/Decode Disaggregation

For workloads with highly variable traffic spikes, Prefill/Decode (P/D) Disaggregation decouples the compute-heavy prompt processing (Prefill) from the memory-bandwidth-bound token generation (Decode) onto separate dedicated GPU pools.

# Use UCX, the documented default
export TRTLLM_NIXL_KVCACHE_BACKEND=UCX

# Explicitly force RDMA (RoCEv2/InfiniBand) to prevent TCP fallback
export TRTLLM_NIXL_USE_RDMA=1

An unsupported backend value falls back to UCX according to documented behavior. Do not assume that choosing UCX automatically guarantees RDMA operation. Without explicitly setting TRTLLM_NIXL_USE_RDMA=1, the network layer will silently fall back to standard TCP, destroying sub-millisecond transfer speeds. Validate the active backend using runtime logs and verify the actual network transfer path with appropriate system and network diagnostics.

5. Diagnosing Host-Memory Exhaustion During Compilation

During ModelOpt quantization or Triton graph compilation, SREs sometimes encounter LLVM ERROR: out of memory crashes.

This error often represents host-side RAM exhaustion during compilation, not a GPU VRAM limit. Heavy swapping to an NVMe drive can substantially slow compilation. To diagnose and mitigate this:

• Identify the relevant build system and inspect the failing stage logs.
• If the host RAM is exhausted by concurrent compilation, reduce the supported worker concurrency. For example, if the build stage honors the setting, restrict the jobs:

export MAX_JOBS=4

If GPU VRAM is exhausted during model loading, adjust GPU memory requirements, parallelism, or the workload configuration.

6. Complete Deployment Workflow

The official NVIDIA container workflow requires starting the context (Prefill) and generation (Decode) workers, followed by the orchestrator.

First, start the container interactively. Crucially, use --network=host instead of standard port binding (-p 8000:8000). Without host network access, internal ports like 5001 and 5002 cannot route KV Cache transfers across multiple nodes via the RDMA/RoCEv2 fabric.

docker run --rm -it --ipc=host --network=host --gpus all \
  --ulimit memlock=-1 --ulimit stack=67108864 \
  -e TRTLLM_NO_USAGE_STATS=1 \
  -v /mnt/shared/models:/models \
  nvcr.io/nvidia/tensorrt-llm/release:1.4.0rc2 /bin/bash

Inside the container, initialize the Context and Generation nodes, then bind them with the Orchestrator (execute these in separate shell sessions or via a process manager):

# 1. Fire up the Generation Node (Decode)
trtllm-serve --model_dir /models/decode_engine/ --type generation --port 5001

# 2. Fire up the Context Node (Prefill)
trtllm-serve --model_dir /models/prefill_engine/ --type context --port 5002

# 3. Launch the Orchestrator via configuration file
trtllm-serve disaggregated -c /models/disagg_config.yaml

7. Production Security and Compliance

Enterprise deployments must address infrastructure security and compliance before exposing APIs.

• Telemetry Controls for Sensitive Deployments: The 1.3+ release line collects anonymous deployment and runtime metadata by default. While this does not capture proprietary prompts or weights, organizations with strict policies should disable it: export TRTLLM_NO_USAGE_STATS=1.
• Public API Exposure: Binding trtllm-serve to 0.0.0.0 means the service listens on all available network interfaces. Do not expose an inference endpoint publicly until you have verified the application's security controls (Authentication, TLS termination, network segmentation, and rate limiting).
• Data Sovereignty and Compliance: For sensitive workloads, evaluate the legal jurisdiction of relevant providers, data residency, international data transfers, and contractual protections. Dedicated bare-metal infrastructure can offer greater control over hardware allocation and deployment location, but compliance must be assessed against the actual processing arrangement and applicable legal requirements.

8. GPU Hardware Sizing

A model's hardware requirements cannot be established from parameter count alone. You must account for weight precision, KV cache limits, context lengths, and parallelism strategy. The table below provides illustrative infrastructure classes, not verified minimum requirements.

• Small Dense Models: One compatible GPU (e.g., L40S, H200). Must validate weight memory, KV cache, concurrency, and latency.
• Larger Dense Models: Multiple high-memory GPUs. Must validate GPU memory budget, TP requirements, and throughput.
• MoE Models: Multiple GPUs with suitable intra-node or inter-node interconnects (e.g., NVSwitch). Must validate expert placement, communication overhead, and supported parallelism.
• Frontier-Scale MoE: Multi-GPU node or rack-scale system (e.g., GB200 NVL72). Must validate total memory, network topology, EP/TP/PP configuration, and benchmarks.

9. Benchmarking and Observability

A high-throughput result under one configuration does not prove universal superiority. For a meaningful comparison against engines like vLLM, you must benchmark using:

• Identical model checkpoints, numerical formats, and GPU topologies.
• Identical input sequence and output sequence distributions.
• Measured Time to First Token (TTFT), Inter-Token Latency (ITL), and aggregate output throughput.

Validate the active backend using runtime logs, and benchmark KV cache transfer latency under representative loads before making production performance claims.