> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/TensorRT-LLM/llms.txt
> Use this file to discover all available pages before exploring further.

# Production Deployment

> Best practices for deploying TensorRT-LLM in production environments

## Overview

This guide covers best practices, performance tuning, monitoring, and scaling strategies for production TensorRT-LLM deployments.

## Architecture Considerations

### Choosing a Backend

> **PyTorch Backend**
>
> **Default choice**
>
> * Best compatibility
> * Active development
> * Full feature support
> * Easier debugging

> **TensorRT Backend**
>
> **Maximum performance**
>
> * Lowest latency
> * Highest throughput
> * Requires build step
> * Limited to specific models

> **AutoDeploy Backend**
>
> **Experimental**
>
> * Automatic optimization
> * On-the-fly quantization
> * Beta stability

> [!TIP]
>
> Start with **PyTorch backend** for development and production. Switch to **TensorRT backend** only if you need absolute maximum performance and your model is supported.

### Deployment Patterns

**Single Server**

Best for:

* Low to medium traffic
* Development/testing
* Models under 70B parameters on single GPU

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --max_batch_size 128 \
  --max_num_tokens 8192
```

**Load Balanced Fleet**

Best for:

* High traffic
* Horizontal scaling
* Fault tolerance

Deploy multiple instances behind NGINX/HAProxy:

```bash theme={null}
# Instance 1 (GPU 0)
CUDA_VISIBLE_DEVICES=0 trtllm-serve model --port 8001 &

# Instance 2 (GPU 1)
CUDA_VISIBLE_DEVICES=1 trtllm-serve model --port 8002 &

# Load balancer distributes requests
```

**Disaggregated**

Best for:

* Long prompts + short outputs
* Optimizing TTFT and TPOT separately
* Large-scale deployments

See [Distributed Inference](/deployment/distributed-inference.md#disaggregated-serving) for details.

## Performance Tuning

### Memory Management

**1. Optimize KV cache allocation**

```yaml config.yaml theme={null}
kv_cache_config:
  free_gpu_memory_fraction: 0.95  # Use 95% of free memory
  enable_block_reuse: true        # Reuse cache blocks
  tokens_per_block: 32            # Larger blocks = less overhead
```

> [!WARNING]
>
> Setting `free_gpu_memory_fraction` too high may cause OOM errors during traffic spikes.

**2. Enable FP8 KV cache**

```yaml theme={null}
kv_cache_config:
  dtype: fp8  # 2x memory savings vs FP16
```

> [!NOTE]
>
> FP8 KV cache requires Hopper GPUs (H100, H200) for best performance.

**3. Monitor cache efficiency**

```bash theme={null}
curl http://localhost:8000/metrics | jq '.[] | .kvCacheStats'
```

Target metrics:

* **Cache hit rate**: >50% for similar prompts
* **Free blocks**: >10% of max blocks
* **Tokens per block**: 32 (default)

### Batching Configuration

```yaml config.yaml theme={null}
max_batch_size: 256        # Max concurrent requests
max_num_tokens: 16384      # Max tokens across batch

scheduler_config:
  capacity_scheduler_policy: GUARANTEED_NO_EVICT
  dynamic_batch_config:
    enable_batch_size_tuning: true
    enable_max_num_tokens_tuning: false
    dynamic_batch_moving_average_window: 128
```

> **Tuning Guidelines**
>
> | Workload | `max_batch_size` | `max_num_tokens` | Notes |
> | - | - | - | - |
> | Short prompts + outputs | 512 | 8192 | Maximize throughput |
> | Long prompts | 128 | 32768 | Prevent OOM |
> | Streaming responses | 256 | 16384 | Balance latency/throughput |
> | Mixed workload | 256 | 16384 | Safe defaults |

### CUDA Graphs

Enable CUDA graphs for 20-30% latency reduction on decode steps:

```yaml theme={null}
cuda_graph_config:
  enable_padding: true
  batch_sizes: [1, 2, 4, 8, 16, 32, 64, 128]  # Pre-capture these sizes
```

> [!WARNING]
>
> CUDA graphs increase memory usage (\~500MB per batch size). Reduce `batch_sizes` if memory-constrained.

### Overlap Scheduler

Enable compute/communication overlap (PyTorch backend only):

```yaml theme={null}
pytorch_backend_config:
  enable_overlap_scheduler: true
```

> [!TIP]
>
> Overlap scheduler can improve throughput by 10-15% for multi-GPU deployments.

## Monitoring and Observability

### Metrics to Track

> **Request Metrics**
>
> * **Time to First Token (TTFT)**: Prefill latency
> * **Time Per Output Token (TPOT)**: Decode latency
> * **Request throughput**: Requests/second
> * **Token throughput**: Tokens/second
> * **Queue time**: Time waiting in scheduler

> **System Metrics**
>
> * **GPU utilization**: Target >80%
> * **GPU memory usage**: Monitor for OOM
> * **KV cache hit rate**: Higher = better efficiency
> * **Active requests**: Current concurrency
> * **Batch sizes**: Average batch utilization

> **Error Metrics**
>
> * **Request failures**: HTTP 5xx errors
> * **OOM errors**: KV cache exhaustion
> * **Timeout errors**: Requests exceeding max wait time

### Collecting Metrics

```python Python Client theme={null}
import requests
import time

def monitor_metrics(interval=5):
    while True:
        response = requests.get("http://localhost:8000/metrics")
        stats = response.json()
        
        for stat in stats:
            print(f"Iter {stat['iter']}:")
            print(f"  GPU Memory: {stat['gpuMemUsage'] / 1e9:.2f} GB")
            print(f"  KV Cache Hit Rate: {stat['kvCacheStats']['cacheHitRate']:.2%}")
            print(f"  Active Requests: {stat['numActiveRequests']}")
            print(f"  Iteration Latency: {stat['iterLatencyMS']:.2f}ms")
        
        time.sleep(interval)

monitor_metrics()
```

```bash cURL + jq theme={null}
#!/bin/bash

while true; do
  curl -s http://localhost:8000/metrics | jq -r '.[] | 
    "Iter: \(.iter) | 
     GPU: \(.gpuMemUsage / 1000000000 | round)GB | 
     Cache Hit: \(.kvCacheStats.cacheHitRate * 100 | round)% | 
     Active: \(.numActiveRequests)"'
  sleep 5
done
```

### OpenTelemetry Integration

Export traces to observability platforms:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --otlp_traces_endpoint http://localhost:4318/v1/traces
```

> [!NOTE]
>
> Requires OpenTelemetry Collector running. See [OpenTelemetry docs](https://opentelemetry.io/docs/collector/) for setup.

## Scaling Strategies

### Vertical Scaling (Single Node)

**1. Use tensor parallelism for large models**

```bash theme={null}
# Split Llama-70B across 4 GPUs
trtllm-serve meta-llama/Llama-3.1-70B-Instruct \
  --tp_size 4
```

**2. Increase batch size and token limits**

```yaml theme={null}
max_batch_size: 512
max_num_tokens: 32768
kv_cache_config:
  free_gpu_memory_fraction: 0.98
```

**3. Enable all optimizations**

```yaml theme={null}
cuda_graph_config:
  enable_padding: true
pytorch_backend_config:
  enable_overlap_scheduler: true
kv_cache_config:
  enable_block_reuse: true
  dtype: fp8
```

### Horizontal Scaling (Multi-Instance)

```nginx NGINX Config theme={null}
upstream trtllm_backend {
    least_conn;  # Route to instance with fewest connections
    server localhost:8001;
    server localhost:8002;
    server localhost:8003;
    server localhost:8004;
}

server {
    listen 80;
    
    location / {
        proxy_pass http://trtllm_backend;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        
        # Timeouts for long generations
        proxy_read_timeout 300s;
        proxy_send_timeout 300s;
    }
}
```

```yaml Docker Compose theme={null}
version: '3.8'

services:
  trtllm-1:
    image: nvcr.io/nvidia/tensorrt-llm:latest
    command: trtllm-serve meta-llama/Llama-3.1-8B-Instruct --port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['0']
              capabilities: [gpu]
  
  trtllm-2:
    image: nvcr.io/nvidia/tensorrt-llm:latest
    command: trtllm-serve meta-llama/Llama-3.1-8B-Instruct --port 8000
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              device_ids: ['1']
              capabilities: [gpu]
  
  nginx:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - trtllm-1
      - trtllm-2
```

### Multi-Node Scaling

For models >70B parameters, use multi-node deployment:

```bash theme={null}
# config.yml
cat > config.yml <<EOF
enable_attention_dp: true
pytorch_backend_config:
  enable_overlap_scheduler: true
EOF

# Slurm deployment (2 nodes, 16 GPUs total)
srun -N 2 \
  --ntasks 16 --ntasks-per-node=8 \
  --mpi=pmix --gres=gpu:8 \
  --container-image=nvcr.io/nvidia/tensorrt-llm:latest \
  bash -c "trtllm-llmapi-launch trtllm-serve deepseek-ai/DeepSeek-V3 \
    --tp_size 16 \
    --ep_size 4 \
    --max_batch_size 161 \
    --config ./config.yml"
```

## Security Best Practices

**1. Disable trust\_remote\_code in production**

```bash theme={null}
trtllm-serve <model>  # trust_remote_code defaults to False
```

> [!WARNING]
>
> Only enable `--trust_remote_code` for models from trusted sources.

**2. Use authentication/authorization**

Deploy behind API gateway with auth:

```nginx theme={null}
location /v1 {
    auth_request /auth;
    proxy_pass http://trtllm_backend;
}

location = /auth {
    internal;
    proxy_pass http://auth-service/verify;
}
```

**3. Rate limiting**

```nginx theme={null}
limit_req_zone $binary_remote_addr zone=api_limit:10m rate=10r/s;

location /v1 {
    limit_req zone=api_limit burst=20;
    proxy_pass http://trtllm_backend;
}
```

**4. Input validation**

Set reasonable limits:

```yaml theme={null}
max_seq_len: 8192      # Prevent excessive memory use
max_tokens: 2048       # Limit output length
```

## High Availability

### Health Checks

```python theme={null}
import requests
import time

def health_check(url: str, timeout: int = 5) -> bool:
    try:
        response = requests.get(f"{url}/health", timeout=timeout)
        return response.status_code == 200
    except:
        return False

# Kubernetes liveness probe
while True:
    if not health_check("http://localhost:8000"):
        print("Service unhealthy, restarting...")
        # Trigger restart
    time.sleep(30)
```

### Graceful Shutdown

```python theme={null}
import signal
import sys
from tensorrt_llm import LLM

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")

def shutdown_handler(signum, frame):
    print("Shutting down gracefully...")
    llm.shutdown()  # Finish in-flight requests
    sys.exit(0)

signal.signal(signal.SIGTERM, shutdown_handler)
signal.signal(signal.SIGINT, shutdown_handler)

# Start serving...
```

### Kubernetes Deployment

```yaml trtllm-deployment.yaml theme={null}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trtllm-serve
spec:
  replicas: 3
  selector:
    matchLabels:
      app: trtllm
  template:
    metadata:
      labels:
        app: trtllm
    spec:
      containers:
      - name: trtllm
        image: nvcr.io/nvidia/tensorrt-llm:latest
        command: ["trtllm-serve"]
        args:
          - "meta-llama/Llama-3.1-8B-Instruct"
          - "--port=8000"
          - "--config=/config/config.yaml"
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 32Gi
          requests:
            nvidia.com/gpu: 1
            memory: 16Gi
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 30
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 30
          periodSeconds: 10
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: trtllm-config
---
apiVersion: v1
kind: Service
metadata:
  name: trtllm-service
spec:
  selector:
    app: trtllm
  ports:
  - port: 80
    targetPort: 8000
  type: LoadBalancer
```

## Troubleshooting

> **Out of Memory (OOM) Errors**
>
> **Symptoms**: Requests fail with CUDA OOM errors
>
> **Solutions**:
>
> 1. Reduce `max_batch_size` or `max_num_tokens`
> 2. Lower `free_gpu_memory_fraction` to 0.85
> 3. Enable FP8 KV cache: `kv_cache_config.dtype: fp8`
> 4. Disable CUDA graphs if enabled
> 5. Use tensor parallelism for larger models

> **High Latency**
>
> **Symptoms**: TTFT or TPOT higher than expected
>
> **Solutions**:
>
> 1. Enable CUDA graphs: `use_cuda_graph: true`
> 2. Enable overlap scheduler (PyTorch): `enable_overlap_scheduler: true`
> 3. Increase `max_num_tokens` to allow larger batches
> 4. Check GPU utilization (should be >80%)
> 5. Reduce `tokens_per_block` to 16 for short requests

> **Low Throughput**
>
> **Symptoms**: Requests/second below expectations
>
> **Solutions**:
>
> 1. Increase `max_batch_size`
> 2. Enable KV cache reuse: `enable_block_reuse: true`
> 3. Use async generation: `generate_async()`
> 4. Check queue time in metrics (should be less than 100ms)
> 5. Scale horizontally with load balancer

> **Model Not Loading**
>
> **Symptoms**: Server fails to start, model download issues
>
> **Solutions**:
>
> 1. Check HuggingFace token: `huggingface-cli login`
> 2. Pre-download model: `huggingface-cli download <model>`
> 3. Verify disk space (models can be 50GB+)
> 4. Check model compatibility with backend
> 5. Enable `--trust_remote_code` if using custom model code

## Performance Checklist

**1. Choose optimal backend**

✅ PyTorch for compatibility, TensorRT for max performance

**2. Configure KV cache**

✅ `free_gpu_memory_fraction: 0.95`
✅ `enable_block_reuse: true`
✅ `dtype: fp8` (on Hopper GPUs)

**3. Set batch limits**

✅ `max_batch_size: 256` (adjust for GPU memory)
✅ `max_num_tokens: 16384`

**4. Enable optimizations**

✅ CUDA graphs
✅ Overlap scheduler (PyTorch)
✅ Dynamic batching

**5. Monitor metrics**

✅ GPU utilization >80%
✅ KV cache hit rate >50%
✅ Queue time less than 100ms

**6. Scale appropriately**

✅ Tensor parallelism for large models
✅ Horizontal scaling for high traffic
✅ Load balancer for multi-instance

## Next Steps

> **[Distributed Inference](/deployment/distributed-inference.md)**
>
> Multi-GPU and multi-node deployments

> **[Benchmarking](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/developer-guide/perf-benchmarking.md)**
>
> Measure and optimize performance

> **[Reference Configs](https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/configs/database)**
>
> 170+ optimized configurations

> **[API Reference](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html)**
>
> Complete configuration reference


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.