> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/TensorRT-LLM/llms.txt
> Use this file to discover all available pages before exploring further.

# trtllm-bench

> Benchmark TensorRT-LLM models for throughput and latency performance

Benchmark TensorRT-LLM models for throughput and latency performance. Supports multiple backends (PyTorch, TensorRT, AutoDeploy) and various benchmarking modes.

## Usage

```bash theme={null}
trtllm-bench --model MODEL [OPTIONS] COMMAND
```

## Global Options

**--model** `string` *required*

The HuggingFace name of the model to benchmark. Alias: `-m`

**--model\_path** `string`

Path to a HuggingFace checkpoint directory for loading model components

**--workspace** `string` (default: /tmp)

The directory to store benchmarking intermediate files. Alias: `-w`

**--log\_level** `string` (default: info)

The logging level. Choices: `verbose`, `info`, `warning`, `error`, `internal_error`

**--revision** `string`

The revision to use for the HuggingFace model (branch name, tag name, or commit id)

## Commands

### throughput

Run a throughput benchmark to measure maximum request processing capacity.

```bash theme={null}
trtllm-bench --model MODEL throughput [OPTIONS]
```

#### Throughput Options

**--backend** `string` (default: pytorch)

Backend to use for benchmark. Choices: `pytorch`, `tensorrt`, `_autodeploy`

**--engine\_dir** `path`

Path to a serialized TRT-LLM engine (for TensorRT backend)

**--dataset** `path` *required*

Path to dataset file for benchmarking. Use `prepare-dataset` command to generate

**--num\_requests** `integer` (default: 0)

Number of requests to cap benchmark run at. If not specified or set to 0, it will be the length of dataset

**--warmup** `integer` (default: 2)

Number of requests to warm up benchmark

**--max\_batch\_size** `integer`

Maximum runtime batch size to run the engine with

**--max\_num\_tokens** `integer`

Maximum runtime tokens that an engine can accept

**--max\_seq\_len** `integer`

Maximum sequence length

**--beam\_width** `integer` (default: 1)

Number of search beams

**--kv\_cache\_free\_gpu\_mem\_fraction** `float` (default: 0.9)

The percentage of memory to use for KV Cache after model load

**--tp** `integer` (default: 1)

Tensor parallelism size

**--pp** `integer` (default: 1)

Pipeline parallelism size

**--ep** `integer`

Expert parallelism size (for MoE models)

**--config** `string`

Path to a YAML file that overwrites the parameters. Can be specified as either `--config` or `--extra_llm_api_options`

**--sampler\_options** `path`

Path to a YAML file that sets sampler options

**--no\_skip\_tokenizer\_init** `boolean` (default: false)

Do not skip tokenizer initialization when loading the model (required for VLM models)

**--eos\_id** `integer` (default: -1)

Set the end-of-sequence token for the benchmark. Set to -1 to disable EOS

**--modality** `string`

Modality of the multimodal requests. Choices: `image`, `video`

**--image\_data\_format** `string` (default: pt)

Format of the image data for multimodal models. Choices: `pt`, `pil`

**--data\_device** `string` (default: cuda)

Device to load the multimodal data on. Choices: `cuda`, `cpu`

**--max\_input\_len** `integer` (default: 4096)

Maximum input sequence length to use for multimodal models

**--target\_input\_len** `integer`

Target (average) input length for tuning heuristics

**--target\_output\_len** `integer`

Target (average) sequence length for tuning heuristics

**--custom\_module\_dirs** `path`

Paths to custom module directories to import

### latency

Run a latency benchmark to measure per-request response times.

```bash theme={null}
trtllm-bench --model MODEL latency [OPTIONS]
```

#### Latency Options

**--backend** `string` (default: pytorch)

Backend to use for benchmark. Choices: `pytorch`, `tensorrt`, `_autodeploy`

**--engine\_dir** `path`

Path to a serialized TRT-LLM engine

**--dataset** `path`

Path to dataset file for benchmarking

**--num\_requests** `integer` (default: 0)

Number of requests to cap benchmark run at. Minimum between value and length of dataset

**--warmup** `integer` (default: 2)

Number of requests to warm up benchmark

**--concurrency** `integer` (default: 1)

Desired concurrency rate (number of requests processing at the same time), less than or equal to 0 for no concurrency limit

**--beam\_width** `integer` (default: 1)

Number of search beams

**--kv\_cache\_free\_gpu\_mem\_fraction** `float` (default: 0.9)

The percentage of memory to use for KV Cache after model load

**--max\_seq\_len** `integer`

Maximum sequence length

**--tp** `integer` (default: 1)

Tensor parallelism size

**--pp** `integer` (default: 1)

Pipeline parallelism size

**--ep** `integer`

Expert parallelism size

**--config** `string`

Path to a YAML file that overwrites the parameters

**--sampler\_options** `path`

Path to a YAML file that sets sampler options

**--modality** `string`

Modality of the multimodal requests. Choices: `image`, `video`

**--max\_input\_len** `integer` (default: 4096)

Maximum input sequence length to use for multimodal models

**--medusa\_choices** `path`

Path to a YAML file that defines the Medusa tree (for speculative decoding)

**--report\_json** `path`

Path where report should be written to

**--iteration\_log** `path`

Path where iteration logging is written to

### prepare-dataset

Prepare a dataset for benchmarking.

```bash theme={null}
trtllm-bench --model MODEL prepare-dataset [OPTIONS]
```

### build

Build a TensorRT engine for benchmarking.

```bash theme={null}
trtllm-bench --model MODEL build [OPTIONS]
```

## Examples

### Throughput Benchmark

Run a throughput benchmark with a prepared dataset:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  throughput \
  --dataset dataset.json \
  --backend pytorch
```

### Latency Benchmark

Measure single-request latency:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  latency \
  --dataset dataset.json \
  --concurrency 1
```

### Multi-GPU Throughput

Benchmark with tensor parallelism:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-70B-Instruct \
  throughput \
  --dataset dataset.json \
  --tp 4 \
  --max_batch_size 256
```

### TensorRT Engine Benchmark

Benchmark a pre-built TensorRT engine:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  throughput \
  --engine_dir ./engine_outputs \
  --dataset dataset.json \
  --backend tensorrt
```

### With Custom Configuration

Use a YAML configuration file:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  throughput \
  --dataset dataset.json \
  --config bench_config.yaml
```

### Prepare Dataset

Generate a benchmark dataset:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  prepare-dataset \
  --output dataset.json \
  --num-requests 1000
```

### High Concurrency Latency Test

Test latency under load:

```bash theme={null}
trtllm-bench --model meta-llama/Llama-3.1-8B-Instruct \
  latency \
  --dataset dataset.json \
  --concurrency 32 \
  --num_requests 500
```

### Multimodal Benchmark

Benchmark a vision-language model:

```bash theme={null}
trtllm-bench --model llava-hf/llava-1.5-7b-hf \
  throughput \
  --dataset vlm_dataset.json \
  --modality image \
  --no_skip_tokenizer_init
```

## Related Commands

* [trtllm-serve](/cli/trtllm-serve) - Serve models via API
* [trtllm-build](/cli/trtllm-build) - Build TensorRT engines
* [trtllm-eval](/cli/trtllm-eval) - Evaluate model accuracy


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.