Optimization and Tuning

Optimization and Tuning#

This guide aims to help users improve vLLM-Ascend performance at the system level. It includes OS configuration, library optimization, deployment guide, and so on. Any feedback is welcome.

Preparation#

Run the container:

# Update DEVICE according to your device (/dev/davinci[0-7])
export DEVICE=/dev/davinci0
# Update the cann base image
export IMAGE=m.daocloud.io/quay.io/ascend/cann:8.5.0-910b-ubuntu22.04-py3.11
docker run --rm \
--name performance-test \
--shm-size=1g \
--device $DEVICE \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash

Configure your environment:

# Configure the mirror
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" > /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-updates main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-backports main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list && \
echo "deb-src https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ jammy-security main restricted universe multiverse" >> /etc/apt/sources.list

# Install os packages
apt update && apt install wget gcc g++ libnuma-dev git vim -y

Install vllm and vllm-ascend:

# Install necessary dependencies
pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
pip install modelscope pandas datasets gevent sacrebleu rouge_score pybind11 pytest

# Configure this var to speed up model download
VLLM_USE_MODELSCOPE=true

Please follow the Installation Guide to make sure vLLM and vllm-ascend are installed correctly.

Note

Make sure your vLLM and vllm-ascend are installed after your Python configuration is completed, because these packages will build binary files using python in current environment. If you install vLLM and vllm-ascend before completing section 1.1, the binary files will not use the optimized python.

Optimizations#

1. Compilation Optimization#

1.1. Install optimized python#

Python supports LTO and PGO optimization starting from version 3.6 and above, which can be enabled at compile time. And we have offered optimized python packages directly to users for the sake of convenience. You can also reproduce the python build following this tutorial according to your specific scenarios.

mkdir -p /workspace/tmp
cd /workspace/tmp

# Download prebuilt lib and packages
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libcrypto.so.1.1
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libomp.so
wget https://repo.oepkgs.net/ascend/pytorch/vllm/lib/libssl.so.1.1
wget https://repo.oepkgs.net/ascend/pytorch/vllm/python/py311_bisheng.tar.gz

# Configure python and pip
cp ./*.so* /usr/local/lib
tar -zxvf ./py311_bisheng.*  -C /usr/local/
mv  /usr/local/py311_bisheng/  /usr/local/python
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3
sed -i "1c#\!/usr/local/python/bin/python3.11" /usr/local/python/bin/pip3.11
ln -sf  /usr/local/python/bin/python3  /usr/bin/python
ln -sf  /usr/local/python/bin/python3  /usr/bin/python3
ln -sf  /usr/local/python/bin/python3.11  /usr/bin/python3.11
ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip3
ln -sf  /usr/local/python/bin/pip3  /usr/bin/pip

export PATH=/usr/bin:/usr/local/python/bin:$PATH

2. OS Optimization#

2.1. jemalloc#

jemalloc is a memory allocator that improves performance for multi-threaded scenarios and can reduce memory fragmentation. jemalloc uses a local thread memory manager to allocate variables, which can avoid lock competition between threads and can hugely optimize performance.

# Install jemalloc
sudo apt update
sudo apt install libjemalloc2

# Configure jemalloc
export LD_PRELOAD=/usr/lib/"$(uname -i)"-linux-gnu/libjemalloc.so.2 $LD_PRELOAD

2.2. Tcmalloc#

TCMalloc (Thread Caching Malloc) is a universal memory allocator that improves overall performance while ensuring low latency by introducing a multi-level cache structure, reducing mutex contention and optimizing large object processing flow. Find more details here.

# Install tcmalloc
sudo apt update
sudo apt install libgoogle-perftools4 libgoogle-perftools-dev

# Get the location of libtcmalloc.so*
find /usr -name libtcmalloc.so*

# Make the priority of tcmalloc higher
# The <path> is the location of libtcmalloc.so we get from the upper command
# Example: "$LD_PRELOAD:/usr/lib/aarch64-linux-gnu/libtcmalloc.so"
export LD_PRELOAD="$LD_PRELOAD:<path>"

# Verify your configuration
# The path of libtcmalloc.so will be contained in the result if your configuration is valid
ldd `which python`

3. torch_npu Optimization#

Some performance tuning features in torch_npu are controlled by environment variables. Some features and their related environment variables are shown below.

Memory optimization:

# Upper limit of memory block splitting allowed (MB): Setting this parameter can prevent large memory blocks from being split.
export PYTORCH_NPU_ALLOC_CONF="max_split_size_mb:250"

# When operators on the communication stream have dependencies, they all need to be ended before being released for reuse. The logic of multi-stream reuse is to release the memory on the communication stream in advance so that the computing stream can be reused.
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

Scheduling optimization:

# Optimize operator delivery queue. This will affect the memory peak value, and may degrade if the memory is tight.
export TASK_QUEUE_ENABLE=2

# This will greatly improve the CPU bottleneck model and ensure the same performance for the NPU bottleneck model.
export CPU_AFFINITY_CONF=1

4. CANN Optimization#

4.1. HCCL Optimization#

There are some performance tuning features in HCCL, which are controlled by environment variables.

You can configure HCCL to use “AIV” mode to optimize performance by setting the environment variable shown below. In “AIV” mode, the communication is scheduled by AI vector core directly with RoCE, instead of being scheduled by AI CPU.

export HCCL_OP_EXPANSION_MODE="AIV"

Plus, there are more features for performance optimization in specific scenarios, which are shown below.

  • HCCL_INTRA_ROCE_ENABLE: Use RDMA link instead of SDMA link between two 8Ps as the mesh interconnect link. Find more details here.

  • HCCL_RDMA_TC: Use this var to configure traffic class of RDMA NIC. Find more details here.

  • HCCL_RDMA_SL: Use this var to configure service level of RDMA NIC. Find more details here.

  • HCCL_BUFFSIZE: Use this var to control the cache size for sharing data between two NPUs. Find more details here.

5. OS Optimization#

This section describes operating system–level optimizations applied on the host machine (bare metal or Kubernetes node) to improve performance stability, latency, and throughput for inference workloads.

Note

These settings must be applied on the host OS and with root privileges. Not inside containers.

5.1#

Set CPU Frequency Governor to performance

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Purpose

  • Forces all CPU cores to run under the performance governor

  • Disables dynamic frequency scaling (e.g., ondemand, powersave)

Benefits

  • Keeps CPU cores at maximum frequency

  • Reduces latency jitter

  • Improves predictability for inference workloads

5.2 Disable Swap Usage#

sysctl -w vm.swappiness=0

Purpose

  • Minimizes the kernel’s tendency to swap memory pages to disk

Benefits

  • Prevents severe latency spikes caused by swapping

  • Improves stability for large in-memory models

Notes

  • For inference workloads, swap can introduce second-level latency

  • Recommended values are 0 or 1

5.3 Disable Automatic NUMA Balancing#

sysctl -w kernel.numa_balancing=0

Purpose

  • Disables the kernel’s automatic NUMA page migration mechanism

Benefits

  • Prevents background memory page migrations

  • Reduces unpredictable memory access latency

  • Improves performance stability on NUMA systems

Recommended For

  • Multi-socket servers

  • Ascend / NPU deployments with explicit NUMA binding

  • Systems with manually managed CPU and memory affinity

5.4 Increase Scheduler Migration Cost#

sysctl -w kernel.sched_migration_cost_ns=50000

Purpose

  • Increases the cost for the scheduler to migrate tasks between CPU cores

Benefits

  • Reduces frequent thread migration

  • Improves CPU cache locality

  • Lowers latency jitter for inference workloads

Parameter Details

  • Unit: nanoseconds (ns)

  • Typical recommended range: 50000–100000

  • Higher values encourage threads to stay on the same CPU core

Contents