A100八卡部署Qwen3.5-122B-A10B实战：Docker与裸机vLLM启动、FP8优化与Benchmark

发表于 | 分类于 | 1.1k | 6 分钟

Docker 启动命令—Qwen3-235B-A22B-Instruct-2507

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
sudo docker run \
  --gpus '"device=0,1,2,3,4,5,6,7"' \
  -v /data/modelRepository/qwen3-repo:/data/modelRepository/qwen3-repo \
  -p 8080:8080 \
  --ipc=host \
  --restart=always \
  -e TZ=UTC \
  harbor-cmp.zoomlion.com/library/vllm/vllm-openai:v0.8.5.post1 \
  --host 0.0.0.0 \
  --port 8080 \
  --max-model-len 131072 \
  --tensor-parallel-size 8 \
  --served-model-name Qwen3-235B-A22B-Instruct-2507 \
  --model /data/modelRepository/qwen3-repo/qwen3-repo_235b-a22b-instruct-2507 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85 \
  --rope-scaling '{"rope_type":"yarn", "factor":4.0,"original_max_position_embeddings":32768}' \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code \
  --api-key df444f1463bd4fc3847fd6c50512f7b

裸机启动命令—Qwen3-235B-A22B-Instruct-2507

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
python3 -m vllm.entrypoints.openai.api_server \
  --host 0.0.0.0 \
  --port 8080 \
  --max-model-len 131072 \
  --tensor-parallel-size 8 \
  --served-model-name Qwen3-235B-A22B-Instruct-2507 \
  --model /data/modelRepository/qwen3-repo/qwen3-repo_235b-a22b-instruct-2507 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.85 \
  --rope-scaling '{"rope_type":"yarn", "factor":4.0,"original_max_position_embeddings":32768}' \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code \
  --api-key df444f1463bd4fc3847fd6c50512f7b4

裸机启动命令—Qwen3.5-122B-A10B

目前Qwen3.5-122B-A10B模型现状

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
python3 -m vllm.entrypoints.openai.api_server \
  --model ${CMP_MAIN_MODEL_PATH_PREFIX} \
  --served-model-name Qwen3.5-122B-A10B \
  --dtype auto \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8080 \
  --trust-remote-code \
  --max-model-len 204800 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.85 \
  --max-num-batched-tokens 30720 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --max-num-seqs 40 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' \
  --api-key df444f1463bd4fc3847fd6c50512f7b4

优化方案

  1. 模型与精度选型：FP8 预量化权重 + bf16 计算精度，是 Ampere 架构下性价比最高的组合 —— 权重显存直接减半，腾出大量空间给 KV 缓存提升并发；同时 A100 原生 BF16 算力拉满，无额外精度损失。
  2. 并行与注意力：TP=8 单实例张量并行 + FLASHINFER 注意力后端，8 卡全 NVLink 互联下通信效率最优；FLASHINFER 针对 Qwen3.5 MoE 架构深度优化，比 FlashAttention-2 计算速度快 10%~20%，显存效率更高。
  3. 显存与 KV 优化：FP8_E5M2 KV 缓存 + 0.92 显存利用率，KV 显存直接减半，是提升并发性价比最高的手段；0.92 的阈值在 A100 上兼顾了显存利用率和稳定性。
  4. 调度与增值优化：分块预填充 + 前缀缓存 + MTP 投机解码，三者分别解决长请求阻塞、公共前缀重复计算、解码单步出 Token 少的问题，是混合负载下提升有效吞吐的关键。
  5. 启动命令
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
python3 -m vllm.entrypoints.openai.api_server \
  --model ${CMP_MAIN_MODEL_PATH_PREFIX} \
  --served-model-name Qwen3.5-122B-A10B-FP8 \
  --dtype bf16 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8080 \
  --trust-remote-code \
  --max-model-len 65536 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.94 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 128 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e5m2 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":4}' \
  --api-key df444f1463bd4fc3847fd6c50512f7b4

Docker 启动命令—Qwen3.5-122B-A10B-FP8（128k长度，激进配置）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
sudo docker run \
  --gpus '"device=0,1,2,3,4,5,6,7"' \
  -v /data/modelRepository/qwen35-repo:/data/modelRepository/qwen35-repo \
  -p 8080:8080 \
  --ipc=host \
  --restart=always \
  -e TZ=UTC \
  harbor-cmp.zoomlion.com/library/vllm/vllm/vllm-openai:v0.22.0-cu129-ubuntu2404 \
  --served-model-name Qwen3.5-122B-A10B-FP8 \
  --dtype bf16 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8080 \
  --trust-remote-code \
  --max-model-len 131072 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 65536 \
  --max-num-seqs 256 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e5m2 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --attention-backend FLASHINFER \
  --api-key df444f1463bd4fc3847fd6c50512f7b4 \
  --mm-encoder-tp-mode data \
  --mm-processor-cache-type shm

Docker 启动命令—Qwen3.5-122B-A10B-FP8（262K 模型 YARN 拓展至 1M）

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
sudo docker run \
  --gpus '"device=0,1,2,3,4,5,6,7"' \
  -v /data/modelRepository/qwen35-repo:/data/modelRepository/qwen35-repo \
  -p 8080:8080 \
  --ipc=host \
  --restart=always \
  -e TZ=UTC \
  harbor-cmp.zoomlion.com/library/vllm/vllm/vllm-openai:v0.22.0-cu129-ubuntu2404 \
  --served-model-name Qwen3.5-122B-A10B-FP8 \
  --dtype bf16 \
  --disable-custom-all-reduce \
  --host 0.0.0.0 \
  --port 8080 \
  --trust-remote-code \
  --max-model-len 1010000 \
  --tensor-parallel-size 8 \
  --gpu-memory-utilization 0.90 \
  --max-num-batched-tokens 32768 \
  --max-num-seqs 64 \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --kv-cache-dtype fp8_e5m2 \
  --hf-overrides '{"text_config": {"rope_parameters": {
      "mrope_interleaved": true,
      "mrope_section": [11, 11, 10],
      "rope_type": "yarn",
      "rope_theta": 10000000,
      "partial_rotary_factor": 0.25,
      "factor": 4.0,
      "original_max_position_embeddings": 262144
  }}}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser qwen3 \
  --attention-backend FLASHINFER \
  --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":4}' \
  --api-key df444f1463bd4fc3847fd6c50512f7b4
  --mm-encoder-tp-mode data

benchmark评估

1
2
3
4
5
6
7
8
9
vllm bench serve \
  --backend openai-chat \
  --endpoint /v1/chat/completions \
  --model Qwen3.5-122B-A10B-FP8 \
  --dataset-name random \
  --random-input-len 2048 \
  --random-output-len 512 \
  --num-prompts 1000 \
  --request-rate 20

服务调用

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import time
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1",
    timeout=3600
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image_url",
                "image_url": {
                    "url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
                }
            },
            {
                "type": "text",
                "text": "Read all the text in the image."
            }
        ]
    }
]

start = time.time()
response = client.chat.completions.create(
    model="Qwen3.5-122B-A10B-FP8",
    messages=messages,
    max_tokens=2048
)
print(f"Response costs: {time.time() - start:.2f}s")
print(f"Generated text: {response.choices[0].message.content}")
本文结束 感谢您的阅读