Quantization Guide#
Model quantization is a technique that reduces model size and computational overhead by lowering the numerical precision of weights and activations, thereby saving memory and improving inference speed.
vLLM Ascend supports multiple quantization methods. This guide provides instructions for using different quantization tools and running quantized models on vLLM Ascend.
Note
You can choose to convert the model yourself or use the quantized model we uploaded. See https://www.modelscope.cn/models/vllm-ascend/Kimi-K2-Instruct-W8A8. Before you quantize a model, ensure sufficient RAM is available.
Quantization Tools#
vLLM Ascend supports models quantized by two main tools: ModelSlim and LLM-Compressor.
1. ModelSlim (Recommended)#
ModelSlim is an Ascend-friendly compression tool focused on acceleration, using compression techniques, and built for Ascend hardware. It includes a series of inference optimization technologies such as quantization and compression, aiming to accelerate large language dense models, MoE models, multimodal understanding models, multimodal generation models, etc.
Installation#
To use ModelSlim for model quantization, install it from its Git repository:
# Install br_release_MindStudio_8.3.0_20261231 version
git clone https://gitcode.com/Ascend/msit.git -b br_release_MindStudio_8.3.0_20261231
cd msit/msmodelslim
bash install.sh
Model Quantization#
The following example shows how to generate W8A8 quantized weights for the Qwen3-MoE model.
Quantization Script:
cd example/Qwen3-MOE
# Support multi-card quantization
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
# Set model and save paths
export MODEL_PATH="/path/to/your/model"
export SAVE_PATH="/path/to/your/quantized_model"
# Run quantization script
python3 quant_qwen_moe_w8a8.py --model_path $MODEL_PATH \
--save_path $SAVE_PATH \
--anti_dataset ../common/qwen3-moe_anti_prompt_50.json \
--calib_dataset ../common/qwen3-moe_calib_prompt_50.json \
--trust_remote_code True
After quantization completes, the output directory will contain the quantized model files.
For more examples, refer to the official examples.
2. LLM-Compressor#
LLM-Compressor is a unified compressed model library for faster vLLM inference.
Installation#
pip install llmcompressor
Model Quantization#
LLM-Compressor provides various quantization scheme examples.
Dense Quantization#
An example to generate W8A8 dynamic quantized weights for dense model:
# Navigate to LLM-Compressor examples directory
cd examples/quantization/llm-compressor
# Run quantization script
python3 w8a8_int8_dynamic.py
MoE Quantization#
An example to generate W8A8 dynamic quantized weights for MoE model:
# Navigate to LLM-Compressor examples directory
cd examples/quantization/llm-compressor
# Run quantization script
python3 w8a8_int8_dynamic_moe.py
For more content, refer to the official examples.
Currently supported quantization types by LLM-Compressor: W8A8 and W8A8_DYNAMIC.
Running Quantized Models#
Once you have a quantized model which is generated by ModelSlim, you can use vLLM Ascend for inference by specifying the --quantization ascend parameter to enable quantization features, while for models quantized by LLM-Compressor, do not need to add this parameter.
Offline Inference#
import torch
from vllm import LLM, SamplingParams
prompts = [
"Hello, my name is",
"The future of AI is",
]
# Set sampling parameters
sampling_params = SamplingParams(temperature=0.6, top_p=0.95, top_k=40)
llm = LLM(model="/path/to/your/quantized_model",
max_model_len=4096,
trust_remote_code=True,
# Set appropriate TP and DP values
tensor_parallel_size=2,
data_parallel_size=1,
# Set an unused port
port=8000,
# Set serving model name
served_model_name="quantized_model",
# Specify `quantization="ascend"` to enable quantization for models quantized by ModelSlim
quantization="ascend")
outputs = llm.generate(prompts, sampling_params)
for output in outputs:
prompt = output.prompt
generated_text = output.outputs[0].text
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
Online Inference#
# Corresponding to offline inference
python -m vllm.entrypoints.api_server \
--model /path/to/your/quantized_model \
--max-model-len 4096 \
--port 8000 \
--tensor-parallel-size 2 \
--data-parallel-size 1 \
--served-model-name quantized_model \
--trust-remote-code \
--quantization ascend