> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/TensorRT-LLM/llms.txt
> Use this file to discover all available pages before exploring further.

# trtllm-serve

> Launch an OpenAI-compatible API server for TensorRT-LLM models

Launch an OpenAI-compatible API server for serving TensorRT-LLM models. Supports PyTorch, TensorRT, and AutoDeploy backends with both HTTP (REST) and gRPC protocols.

## Usage

```bash theme={null}
trtllm-serve MODEL [OPTIONS]
```

## Arguments

**MODEL** `string` *required*

Model name, HuggingFace checkpoint path, or TensorRT engine path

## Server Options

**--host** `string` (default: localhost)

Hostname of the server

**--port** `integer` (default: 8000)

Port of the server

**--backend** `string` (default: pytorch)

Backend to use for serving the model. Choices: `pytorch`, `tensorrt`, `trt` (alias for tensorrt), `_autodeploy`

**--grpc** `boolean` (default: false)

Run gRPC server instead of OpenAI HTTP server. gRPC server accepts pre-tokenized requests and returns raw token IDs

**--served\_model\_name** `string`

The model name used in the API. If not specified, the model path is used. Useful when the model path is long or when you want to expose a custom name to clients

## Tokenizer Options

**--tokenizer** `string`

Path or name of the tokenizer. Defaults to model path if not specified

**--custom\_tokenizer** `string`

Custom tokenizer type: alias (e.g., 'deepseek\_v32') or Python import path (e.g., 'tensorrt\_llm.tokenizer.deepseek\_v32.DeepseekV32Tokenizer')

## Model Configuration

**--max\_batch\_size** `integer` (default: 8)

Maximum number of requests that the engine can schedule

**--max\_num\_tokens** `integer` (default: 2048)

Maximum number of batched input tokens after padding is removed in each batch

**--max\_seq\_len** `integer`

Maximum total length of one request, including prompt and outputs. If unspecified, the value is deduced from the model config

**--max\_beam\_width** `integer` (default: 1)

Maximum number of beams for beam search decoding

## Parallelism Options

**--tensor\_parallel\_size** `integer` (default: 1)

Tensor parallelism size. Alias: `--tp_size`

**--pipeline\_parallel\_size** `integer` (default: 1)

Pipeline parallelism size. Alias: `--pp_size`

**--context\_parallel\_size** `integer` (default: 1)

Context parallelism size. Alias: `--cp_size`

**--moe\_expert\_parallel\_size** `integer`

Expert parallelism size for MoE models. Alias: `--ep_size`

**--moe\_cluster\_parallel\_size** `integer`

Expert cluster parallelism size. Alias: `--cluster_size`

**--gpus\_per\_node** `integer`

Number of GPUs per node. Defaults to automatic detection

## Memory Options

**--free\_gpu\_memory\_fraction** `float` (default: 0.9)

Free GPU memory fraction reserved for KV Cache, after allocating model weights and buffers. Alias: `--kv_cache_free_gpu_memory_fraction`

**--kv\_cache\_dtype** `string` (default: auto)

KV cache quantization dtype for PyTorch backend. Choices: `auto`, `fp8`, `nvfp4`. 'auto' uses checkpoint/model metadata; explicit values force override

## Advanced Options

**--config** `string`

Path to a YAML file that overwrites the parameters specified by trtllm-serve. Can be specified as either `--config` or `--extra_llm_api_options`

**--trust\_remote\_code** `boolean` (default: false)

Flag for HuggingFace transformers to trust remote code

**--revision** `string`

The revision to use for the HuggingFace model (branch name, tag name, or commit id)

**--num\_postprocess\_workers** `integer` (default: 0)

Number of workers to postprocess raw responses to comply with OpenAI protocol

**--enable\_chunked\_prefill** `boolean` (default: false)

Enable chunked prefill

**--reasoning\_parser** `string`

Specify the parser for reasoning models

**--tool\_parser** `string`

Specify the parser for tool models

**--chat\_template** `string`

Specify a custom chat template. Can be a file path or one-liner template string

**--custom\_module\_dirs** `string`

Paths to custom module directories to import

## Disaggregated Serving

**--server\_role** `string`

Server role. Specify this value only if running in disaggregated mode

**--metadata\_server\_config\_file** `string`

Path to metadata server config file for disaggregated serving

**--disagg\_cluster\_uri** `string`

URI of the disaggregated cluster

## Observability

**--otlp\_traces\_endpoint** `string`

Target URL to which OpenTelemetry traces will be sent

**--log\_level** `string` (default: info)

The logging level. Choices: `verbose`, `info`, `warning`, `error`, `internal_error`

## Multimodal Options

**--media\_io\_kwargs** `string`

Keyword arguments for media I/O (JSON string)

**--extra\_visual\_gen\_options** `string`

Path to a YAML file with extra VISUAL\_GEN model options (for diffusion models)

## Runtime Behavior

**--fail\_fast\_on\_attention\_window\_too\_large** `boolean` (default: false)

Exit with runtime error when attention window is too large to fit even a single sequence in the KV cache

## Examples

### Basic HTTP Server

Serve a model from HuggingFace with default settings:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct
```

### Custom Port and Backend

Serve with TensorRT backend on a custom port:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --backend tensorrt \
  --port 8080
```

### Multi-GPU Tensor Parallel

Serve with 4-way tensor parallelism:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor_parallel_size 4
```

### With Configuration File

Use a YAML configuration file for advanced settings:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --config server_config.yaml
```

### gRPC Server

Launch a gRPC server for pre-tokenized requests:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --grpc \
  --port 50051
```

### Custom Model Name

Serve with a custom model name for API responses:

```bash theme={null}
trtllm-serve /path/to/local/model \
  --served_model_name "my-custom-llama"
```

### High-Performance Configuration

Optimize for throughput with increased batch size and memory allocation:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-8B-Instruct \
  --backend tensorrt \
  --max_batch_size 256 \
  --max_num_tokens 8192 \
  --free_gpu_memory_fraction 0.95 \
  --enable_chunked_prefill
```

### Multi-Node Serving

Serve across multiple nodes with tensor and pipeline parallelism:

```bash theme={null}
trtllm-serve meta-llama/Llama-3.1-70B-Instruct \
  --tensor_parallel_size 4 \
  --pipeline_parallel_size 2 \
  --gpus_per_node 8
```

## Related Commands

* [trtllm-bench](/cli/trtllm-bench.md) - Benchmark model performance
* [trtllm-build](/cli/trtllm-build.md) - Build TensorRT engines
* [trtllm-eval](/cli/trtllm-eval.md) - Evaluate model accuracy


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.