> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/TensorRT-LLM/llms.txt
> Use this file to discover all available pages before exploring further.

# OpenAI-Compatible Server

> Deploy TensorRT-LLM with trtllm-serve for production inference

## Overview

The `trtllm-serve` command starts an OpenAI-compatible server that supports REST and gRPC endpoints. It provides the simplest way to deploy TensorRT-LLM models in production with minimal configuration.

> [!NOTE]
>
> `trtllm-serve` supports all three backends: **PyTorch** (default), **TensorRT**, and **AutoDeploy**.

## Quick Start

Start a server with a HuggingFace model:

```bash theme={null}
trtllm-serve TinyLlama/TinyLlama-1.1B-Chat-v1.0
```

The server will start on `localhost:8000` by default.

## Command-Line Options

### Basic Configuration

```bash theme={null}
trtllm-serve <model> \
  --host localhost \
  --port 8000 \
  --backend pytorch \
  --tp_size 2 \
  --max_batch_size 128 \
  --max_num_tokens 8192
```

**model** `string` *required*

Model path or HuggingFace model name (e.g., `meta-llama/Llama-3.1-8B-Instruct`)

**--host** `string` (default: localhost)

Server hostname

**--port** `int` (default: 8000)

Server port

**--backend** `string` (default: pytorch)

Inference backend: `pytorch`, `tensorrt`, or `_autodeploy`

### Parallelism Configuration

**--tp\_size** `int` (default: 1)

Tensor parallelism size (split model across GPUs)

**--pp\_size** `int` (default: 1)

Pipeline parallelism size (split layers across GPUs)

**--ep\_size** `int`

Expert parallelism size for MoE models

### Performance Tuning

**--max\_batch\_size** `int` (default: 256)

Maximum number of concurrent requests

**--max\_num\_tokens** `int` (default: 8192)

Maximum tokens across all requests in a batch

**--max\_seq\_len** `int`

Maximum sequence length (prompt + generation). Auto-detected from model config if not specified.

**--free\_gpu\_memory\_fraction** `float` (default: 0.9)

Fraction of GPU memory to use for KV cache

## OpenAI-Compatible Endpoints

The server exposes these OpenAI-compatible endpoints:

### Chat Completions

```bash cURL theme={null}
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama-1.1B-Chat-v1.0",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Where is New York?"}
    ],
    "max_tokens": 128,
    "temperature": 0.7
  }'
```

```python Python theme={null}
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="EMPTY"  # Not required for local server
)

response = client.chat.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Where is New York?"}
    ],
    max_tokens=128,
    temperature=0.7
)

print(response.choices[0].message.content)
```

### Text Completions

```bash theme={null}
curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "TinyLlama-1.1B-Chat-v1.0",
    "prompt": "The capital of France is",
    "max_tokens": 64,
    "temperature": 0
  }'
```

### Streaming Responses

```python theme={null}
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

stream = client.chat.completions.create(
    model="TinyLlama-1.1B-Chat-v1.0",
    messages=[{"role": "user", "content": "Tell me a story"}],
    max_tokens=512,
    stream=True
)

for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)
```

## Advanced Configuration with YAML

For complex configurations, use a YAML file with `--config`:

```yaml config.yaml theme={null}
max_batch_size: 128
max_num_tokens: 8192

kv_cache_config:
  free_gpu_memory_fraction: 0.95
  enable_block_reuse: true
  dtype: fp8

pytorch_backend_config:
  enable_overlap_scheduler: true

moe_config:
  backend: CUTLASS

cuda_graph_config:
  enable_padding: true
  batch_sizes: [1, 2, 4, 8, 16, 32]
```

Start the server with the config:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct --config config.yaml
```

> [!TIP]
>
> The YAML file mirrors the structure of [`TorchLlmArgs`](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html#tensorrt_llm.llmapi.TorchLlmArgs). All nested configuration classes can be specified.

## Multi-Node Deployment with Slurm

Deploy large models across multiple nodes using Slurm:

```bash theme={null}
cat > config.yml <<EOF
enable_attention_dp: true
pytorch_backend_config:
  enable_overlap_scheduler: true
EOF

srun -N 2 \
  --ntasks 16 --ntasks-per-node=8 \
  --mpi=pmix --gres=gpu:8 \
  --container-image=nvcr.io/nvidia/tensorrt-llm:latest \
  bash -c "trtllm-llmapi-launch trtllm-serve deepseek-ai/DeepSeek-V3 \
    --max_batch_size 161 \
    --max_num_tokens 1160 \
    --tp_size 16 \
    --ep_size 4 \
    --kv_cache_free_gpu_memory_fraction 0.95 \
    --config ./config.yml"
```

> [!NOTE]
>
> `trtllm-llmapi-launch` is a wrapper script that handles MPI initialization for multi-node deployments.

## gRPC Server Mode

For high-performance use cases with external routers (e.g., sgl-router), use gRPC mode:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --port 50051
```

> [!WARNING]
>
> gRPC mode accepts **pre-tokenized** inputs and returns **raw token IDs**. It does not support `--tool_parser`, `--chat_template`, or disaggregated serving.

## Monitoring and Metrics

### Health Endpoint

```bash theme={null}
curl http://localhost:8000/health
```

### Metrics Endpoint

Enable performance metrics in your config:

```yaml theme={null}
enable_iter_perf_stats: true
```

Query runtime statistics:

```bash theme={null}
curl http://localhost:8000/metrics
```

> **Example Metrics Response**
>
> ```json theme={null}
> [
>   {
>     "gpuMemUsage": 76665782272,
>     "iter": 154,
>     "iterLatencyMS": 7.0,
>     "kvCacheStats": {
>       "allocNewBlocks": 3126,
>       "cacheHitRate": 0.00128,
>       "freeNumBlocks": 101253,
>       "maxNumBlocks": 101256,
>       "tokensPerBlock": 32,
>       "usedNumBlocks": 3
>     },
>     "numActiveRequests": 1
>   }
> ]
> ```

> [!WARNING]
>
> Metrics are stored in a queue and **removed once retrieved**. Poll regularly if you need to retain metrics.

## Custom Tokenizers

Use custom tokenizers for specialized models:

```bash theme={null}
trtllm-serve deepseek-ai/DeepSeek-V3 \
  --custom_tokenizer deepseek_v32
```

Or specify a Python import path:

```bash theme={null}
trtllm-serve deepseek-ai/DeepSeek-V3 \
  --custom_tokenizer tensorrt_llm.tokenizer.deepseek_v32.DeepseekV32Tokenizer
```

## Multimodal Models

For vision-language models (VLMs), disable KV cache reuse:

```yaml vlm-config.yaml theme={null}
kv_cache_config:
  enable_block_reuse: false
```

```bash theme={null}
trtllm-serve Qwen/Qwen2-VL-7B-Instruct --config vlm-config.yaml
```

Send multimodal requests:

```python theme={null}
from openai import OpenAI
import base64

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")

with open("image.jpg", "rb") as f:
    image_b64 = base64.b64encode(f.read()).decode()

response = client.chat.completions.create(
    model="Qwen2-VL-7B-Instruct",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_b64}"}}
        ]
    }]
)

print(response.choices[0].message.content)
```

## Production Best Practices

**1. Use YAML configuration files**

Store configurations in version-controlled YAML files instead of long command lines.

**2. Enable KV cache reuse**

Set `enable_block_reuse: true` in `kv_cache_config` for improved throughput with repetitive prompts.

**3. Tune batch size and token limits**

Adjust `max_batch_size` and `max_num_tokens` based on your GPU memory and workload.

**4. Monitor metrics**

Enable `enable_iter_perf_stats` and poll `/metrics` to track GPU utilization and KV cache efficiency.

**5. Use custom model names**

Specify `--served_model_name` to expose a user-friendly model name in the API instead of the path.

## Next Steps

> **[LLM API](/deployment/llm-api.md)**
>
> Use the Python LLM API for programmatic access

> **[Distributed Inference](/deployment/distributed-inference.md)**
>
> Scale to multi-GPU and multi-node deployments

> **[Production Guide](/deployment/production.md)**
>
> Best practices for production deployments

> **[Disaggregated Serving](/deployment/distributed-inference.md#disaggregated-serving)**
>
> Optimize TTFT and TPOT independently


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.