> ## Documentation Index
> Fetch the complete documentation index at: https://mintlify.com/NVIDIA/TensorRT-LLM/llms.txt
> Use this file to discover all available pages before exploring further.

# trtllm-build

> Build optimized TensorRT engines from model checkpoints

Build optimized TensorRT engines from model checkpoints for high-performance inference. This command compiles model checkpoints into TensorRT engines with various optimization options.

## Usage

```bash theme={null}
trtllm-build [OPTIONS]
```

## Input Options

**--checkpoint\_dir** `string`

The directory path that contains TensorRT-LLM checkpoint

**--model\_config** `string`

The file path that saves TensorRT-LLM checkpoint config

**--build\_config** `string`

The file path that saves TensorRT-LLM build config

**--model\_cls\_file** `string`

The file path that defines customized TensorRT-LLM model

**--model\_cls\_name** `string`

The customized TensorRT-LLM model class name

## Output Options

**--output\_dir** `string` (default: engine\_outputs)

The directory path to save the serialized engine files and engine config file

## Engine Configuration

**--max\_batch\_size** `integer` (default: 8)

Maximum number of requests that the engine can schedule

**--max\_input\_len** `integer` (default: 1024)

Maximum input length of one request

**--max\_seq\_len** `integer`

Maximum total length of one request, including prompt and outputs. If unspecified, the value is deduced from the model config. Alias: `--max_decoder_seq_len`

**--max\_beam\_width** `integer` (default: 1)

Maximum number of beams for beam search decoding

**--max\_num\_tokens** `integer` (default: 2048)

Maximum number of batched input tokens after padding is removed in each batch. Currently, the input padding is removed by default

**--opt\_num\_tokens** `integer`

Optimal number of batched input tokens after padding is removed in each batch. It equals to `max_batch_size * max_beam_width` by default. Set this value as close as possible to the actual number of tokens on your workload

**--max\_encoder\_input\_len** `integer` (default: 1024)

Maximum encoder input length for encoder-decoder models. Set `max_input_len` to 1 to start generation from decoder\_start\_token\_id of length 1

**--max\_prompt\_embedding\_table\_size** `integer` (default: 0)

Maximum prompt embedding table size for prompt tuning, or maximum multimodal input size for multimodal models. Setting a value > 0 enables prompt tuning or multimodal input. Alias: `--max_multimodal_len`

## KV Cache Options

**--kv\_cache\_type** `string`

Set KV cache type. Choices: `continuous`, `paged`, `disabled`. For disabled case, KV cache is disabled and only context phase is allowed

**--paged\_kv\_cache** `string`

Deprecated. Enabling this option is equivalent to `--kv_cache_type paged` for transformer based models

## Build Optimization

**--input\_timing\_cache** `string`

The file path to read the timing cache. This option is ignored if the file does not exist

**--output\_timing\_cache** `string`

The file path to write the timing cache

**--profiling\_verbosity** `string` (default: layer\_names\_only)

The profiling verbosity for the generated TensorRT engine. Setting to detailed allows inspecting tactic choices and kernel parameters. Choices: `layer_names_only`, `detailed`, `none`

**--strip\_plan** `boolean` (default: false)

Enable stripping weights from the final TensorRT engine under the assumption that the refit weights are identical to those provided at build time

**--weight\_sparsity** `boolean` (default: false)

Enable weight sparsity

**--weight\_streaming** `boolean` (default: false)

Enable offloading weights to CPU and streaming loading at runtime

**--fast\_build** `boolean` (default: false)

Enable features for faster engine building. This may cause some performance degradation and is currently incompatible with int8/int4 quantization without plugin

## Build Process Control

**--workers** `integer` (default: 1)

The number of workers for building in parallel

**--log\_level** `string` (default: info)

The logging level. Choices: `verbose`, `info`, `warning`, `error`, `internal_error`

**--enable\_debug\_output** `boolean` (default: false)

Enable debug output

**--visualize\_network** `string`

The directory path to export TensorRT Network as ONNX prior to Engine build for debugging

**--dry\_run** `boolean` (default: false)

Run through the build process except the actual Engine build for debugging

**--monitor\_memory** `boolean` (default: false)

Enable memory monitor during Engine build

## Logits Options

**--logits\_dtype** `string`

The data type of logits. Choices: `float16`, `float32`

**--gather\_context\_logits** `boolean` (default: false)

Enable gathering context logits

**--gather\_generation\_logits** `boolean` (default: false)

Enable gathering generation logits (deprecated, use runtime flag instead)

**--gather\_all\_token\_logits** `boolean` (default: false)

Enable both `gather_context_logits` and `gather_generation_logits`

## LoRA Options

**--lora\_dir** `string`

The directory of LoRA weights. If multiple directories are provided, the first one is used for configuration

**--lora\_ckpt\_source** `string` (default: hf)

The source type of LoRA checkpoint. Choices: `hf`, `nemo`

**--lora\_target\_modules** `string`

The target module names that LoRA is applied. Only effective when `lora_plugin` is enabled. Choices: `attn_qkv`, `attn_q`, `attn_k`, `attn_v`, `attn_dense`, `mlp_h_to_4h`, `mlp_4h_to_h`, `mlp_gate`, `cross_attn_qkv`, `cross_attn_q`, `cross_attn_k`, `cross_attn_v`, `cross_attn_dense`, `moe_h_to_4h`, `moe_4h_to_h`, `moe_gate`, `moe_router`

**--max\_lora\_rank** `integer` (default: 64)

Maximum LoRA rank for different LoRA modules. It is used to compute the workspace size of LoRA plugin

## Speculative Decoding Options

**--speculative\_decoding\_mode** `string`

Mode of speculative decoding. Choices: `draft_tokens_external`, `lookahead_decoding`, `medusa`, `explicit_draft_tokens`, `eagle`

**--max\_draft\_len** `integer` (default: 0)

Maximum lengths of draft tokens for speculative decoding target model

## Plugin Configuration

The build command supports various plugin configuration options. Use `--help` to see the full list of plugin options including:

* `--gpt_attention_plugin`
* `--gemm_plugin`
* `--lora_plugin`
* `--context_fmha`
* `--remove_input_padding`
* And many more

## Examples

### Basic Engine Build

Build a basic TensorRT engine from a checkpoint:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs
```

### Custom Batch and Sequence Length

Build with specific batch size and sequence length limits:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --max_batch_size 64 \
  --max_input_len 2048 \
  --max_seq_len 4096
```

### Parallel Build

Build with multiple workers for faster compilation:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --workers 4
```

### Enable Weight Sparsity

Build with weight sparsity optimization:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --weight_sparsity
```

### Build with Timing Cache

Use timing cache for faster subsequent builds:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --input_timing_cache ./timing_cache.bin \
  --output_timing_cache ./timing_cache.bin
```

### Enable LoRA Support

Build engine with LoRA adapter support:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --lora_dir ./lora_weights \
  --max_lora_rank 64 \
  --lora_target_modules attn_qkv attn_dense mlp_h_to_4h mlp_4h_to_h
```

### Fast Build Mode

Enable fast build for quicker iteration during development:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --fast_build
```

### Build with Speculative Decoding

Enable Medusa speculative decoding:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --speculative_decoding_mode medusa \
  --max_draft_len 5
```

### Build for Multimodal Models

Build engine for vision-language models:

```bash theme={null}
trtllm-build --checkpoint_dir ./vlm_checkpoint \
  --output_dir ./engine_outputs \
  --max_prompt_embedding_table_size 4096 \
  --max_batch_size 16
```

### Detailed Profiling

Build with detailed profiling information:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --profiling_verbosity detailed
```

### Debug Build

Build with debugging and visualization:

```bash theme={null}
trtllm-build --checkpoint_dir ./model_checkpoint \
  --output_dir ./engine_outputs \
  --enable_debug_output \
  --visualize_network ./network_visualization \
  --monitor_memory
```

### Build from Model Config

Build using separate model and build config files:

```bash theme={null}
trtllm-build --model_config ./model_config.json \
  --build_config ./build_config.json \
  --output_dir ./engine_outputs
```

## Build Process

The build process involves:

1. **Loading checkpoint** - Reads model weights and configuration
2. **Network construction** - Builds TensorRT network with optimizations
3. **Engine compilation** - Compiles network into optimized TensorRT engine
4. **Serialization** - Saves engine files to output directory

## Output Files

The `--output_dir` will contain:

* `rank*.engine` - Serialized TensorRT engines (one per GPU rank)
* `config.json` - Engine configuration
* Timing cache (if `--output_timing_cache` is specified)

## Performance Tips

* Use `--workers` to parallelize builds across multiple GPUs
* Enable `--weight_sparsity` for sparse models
* Use timing cache for faster subsequent builds
* Set `--max_batch_size` and `--max_num_tokens` based on your workload
* Enable `--fast_build` during development, disable for production

## Related Commands

* [trtllm-serve](https://mintlify.com/NVIDIA/TensorRT-LLM/cli/trtllm-serve.md) - Serve the built engines
* [trtllm-bench](https://mintlify.com/NVIDIA/TensorRT-LLM/cli/trtllm-bench.md) - Benchmark engine performance
* [trtllm-eval](https://mintlify.com/NVIDIA/TensorRT-LLM/cli/trtllm-eval.md) - Evaluate model accuracy


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.