vLLM与SGLang部署Qwen3-235B-A22B实战：启动参数、K8s部署YAML与长提示吞吐量对比

vLLM 部署 Qwen3-235B-A22B-Instruct-2507

启动命令

sudo docker run --gpus '"device=0,1,2,3,4,5,6,7"' \
  -v /data/models/:/data/models/ \
  -p 30001:30001 \
  --ipc=host \
  --restart=always \
  -e TZ=UTC \
  harbor-cmp.zoomlion.com/library/vllm/vllm-openai:v0.10.1.1 \
  --host 0.0.0.0 --port 30001 \
  --max-model-len 131072 \
  --tensor-parallel-size 8 \
  --served-model-name Qwen3-235B-A22B-Instruct-2507 \
  --model /data/models/Qwen3-235B-A22B-Instruct-2507 \
  --enable-chunked-prefill \
  --enable-prefix-caching --gpu-memory-utilization 0.95 \
  --rope-scaling '{"rope_type":"yarn", "factor":4.0,"original_max_position_embeddings":32768}' \
  --enable-auto-tool-choice \
  --tool-call-parser hermes \
  --trust-remote-code \
  --api-key <YOUR_API_KEY>

启动日志（关键行）

text

INFO 08-27 08:58:50 [__init__.py:241] Automatically detected platform cuda.
(APIServer pid=1) INFO 08-27 08:58:52 [api_server.py:1805] vLLM API server version 0.10.1.1
(APIServer pid=1) INFO 08-27 08:58:52 [utils.py:326] non-default args: {'host': '0.0.0.0', 'port': 30001, 'api_key': ['<YOUR_API_KEY>'], 'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'model': '/data/models/Qwen3-235B-A22B-Instruct-2507', 'trust_remote_code': True, 'rope_scaling': {'rope_type': 'yarn', 'factor': 4.0, 'original_max_position_embeddings': 32768}, 'max_model_len': 131072, 'served_model_name': ['Qwen3-235B-A22B-Instruct-2507'], 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.95, 'enable_prefix_caching': True, 'enable_chunked_prefill': True}
(APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
(APIServer pid=1) INFO 08-27 08:59:00 [__init__.py:711] Resolved architecture: Qwen3MoeForCausalLM
(APIServer pid=1) INFO 08-27 08:59:00 [__init__.py:1750] Using max model len 131072
(APIServer pid=1) INFO 08-27 08:59:01 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=2048.
(EngineCore_0 pid=269) INFO 08-27 08:59:07 [core.py:74] Initializing a V1 LLM engine (v0.10.1.1) with config: model='/data/models/Qwen3-235B-A22B-Instruct-2507', ..., dtype=torch.bfloat16, max_seq_len=131072, tensor_parallel_size=8, ..., enable_prefix_caching=True, chunked_prefill_enabled=True, ...
(VllmWorker TP0 pid=402) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
(VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
(VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
(VllmWorker TP0 pid=402) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.05 seconds
(VllmWorker TP0 pid=402) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.566111 seconds
(VllmWorker TP0 pid=402) INFO 08-27 09:00:27 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_0_0/backbone for vLLM's torch.compile
(VllmWorker TP0 pid=402) INFO 08-27 09:02:32 [backends.py:215] Compiling a graph for dynamic shape takes 123.42 s
(VllmWorker TP0 pid=402) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 145.68 s in total
(VllmWorker TP0 pid=402) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.51 GiB
(EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 435,344 tokens
(EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.32x
(EngineCore_0 pid=269) INFO 08-27 09:04:13 [core.py:214] init engine (profile, create kv cache, warmup model) took 248.22 seconds
(APIServer pid=1) INFO 08-27 09:04:14 [api_server.py:1611] Supported_tasks: ['generate']
(APIServer pid=1) WARNING 08-27 09:04:14 [__init__.py:1625] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator.
(APIServer pid=1) INFO 08-27 09:04:14 [serving_responses.py:120] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=1) INFO 08-27 09:04:14 [serving_chat.py:94] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
(APIServer pid=1) INFO 08-27 09:04:14 [api_server.py:1880] Starting vLLM API server 0 on http://0.0.0.0:30001

调用日志

text

(APIServer pid=1) INFO:     10.33.27.93:55840 - "POST /v1/chat/completions HTTP/1.1" 200 OK
(APIServer pid=1) INFO 08-27 09:04:24 [loggers.py:123] Engine 000: Avg prompt throughput: 2.3 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

vLLM 部署 Qwen3-235B-A22B

vLLM Qwen3-235B-A22B 启动

20250430版本支持qwen3思考/非思考模式

1、–enable-prefix-caching、–chunked-prefill-enabled、–use-v2-block-manager默认开启；
2、–enforce-eager、–enable-expert-parallel默认关闭；

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server \
    --served-model-name qwen3-235b-a22b \
    --port 8995 \
    --model /models/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --max-model-len 32000 \
    --gpu-memory-utilization 0.80 \
    --enable-reasoning \
    --reasoning-parser deepseek_r1 \
    --api-key <YOUR_API_KEY>

vLLM Qwen3-235B-A22B 优化

1、–enable-expert-parallel开启专家并行，Qwen3-235B-A22B-FP8的–tensor-parallel-size为4；
2、–enable-reasoning –reasoning-parser deepseek_r1 –enable-auto-tool-choice –tool-call-parser hermes来支持MCP；
3、–max-num-batched-tokens 、–max_num_seqs默认为None；

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server \
    --served-model-name qwen3-235b-a22b \
    --port 8995 \
    --model /models/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --max-model-len 32000 \
    --gpu-memory-utilization 0.80 \
    --enable-reasoning \
    --reasoning-parser deepseek_r1 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --api-key <YOUR_API_KEY>

–cpu-offload-gb：将每张卡应该装载到显存的模型卸载到CPU内存中，这个性能受限于PCIe速度和内存带宽，卸载到内存中的模型每次前向传播都会装载到GPU的显存中。

vLLM 0.8.5日志分析

• vLLM API server version 0.8.5

INFO 04-29 01:18:54 [api_server.py:1044] args: Namespace(host=None, port=8995, ..., api_key='<YOUR_API_KEY>', ..., model='/models/Qwen3-235B-A22B', task='auto', ..., trust_remote_code=False, load_format='auto', dtype='auto', max_model_len=32000, reasoning_parser='deepseek_r1', ..., tensor_parallel_size=8, enable_expert_parallel=False, ..., gpu_memory_utilization=0.9, swap_space=4, ..., enable_prefix_caching=None, use_v2_block_manager=True, ..., served_model_name=['qwen3-235b-a22b'], ..., enable_reasoning=True, ...)

• This model supports multiple tasks: {‘embed’, ‘classify’, ‘generate’, ‘reward’, ‘score’}. Defaulting to ‘generate’.
• Defaulting to use mp for distributed inference
• Chunked prefill is enabled with max_num_batched_tokens=8192.

INFO 04-29 01:19:11 [core.py:58] Initializing a V1 LLM engine (v0.8.5) with config: model='/models/Qwen3-235B-A22B', ..., dtype=torch.bfloat16, max_seq_len=32000, tensor_parallel_size=8, ..., enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, ...

• vLLM is using nccl==2.21.5
• Using Flash Attention backend on V1 engine.
• Using FlashInfer for top-p & top-k sampling.
• Using default completion sampling params from model: {‘temperature’: 0.6, ‘top_k’: 20, ‘top_p’: 0.95}

vLLM 0.8.5调用日志

INFO 05-18 20:11:58 [logger.py:39] Received request chatcmpl-ba69850c020242479e52db988c7834ca: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n你是谁？<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=31978, ...)

思考模式：流式返回的首个token

{
    "id": "chatcmpl-ba69850c020242479e52db988c7834ca",
    "object": "chat.completion.chunk",
    "created": 1747624318,
    "model": "qwen3-235b-a22b",
    "choices": [
        {
            "index": 0,
            "delta": {
                "role": "assistant",
                "content": ""
            },
            "logprobs": null,
            "finish_reason": null
        }
    ]
}

思考模式：非流式返回

{
    "id": "chatcmpl-c85543dd23024d99944f76ff2e511120",
    "object": "chat.completion",
    "created": 1747624579,
    "model": "qwen3-235b-a22b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "reasoning_content": "\n\n好的，用户问"你是谁？"。。。\n",
                "content": "\n\n你好！我是通义千问。。。。",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 22,
        "total_tokens": 324,
        "completion_tokens": 302,
        "prompt_tokens_details": null
    },
    "prompt_logprobs": null
}

vLLM 使用 Kubernetes 部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-235b
  namespace: custom-model-deploy
  labels:
    app: qwen3-235b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen3-235b
  template:
    metadata:
      labels:
        app: qwen3-235b
    spec:
      volumes:
      # vLLM needs to access the host's shared memory for tensor parallel inference.
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: "8Gi"
        hostPath:
          path: /data/models/Qwen3-235B-A22B
          type: ''
        name: model-vol
      containers:
      - name: qwen3-235b
        image: vllm/vllm-openai:v0.8.5.post1
        command: ["/bin/sh", "-c"]
        args: [
          "python3 serve mistralai/Mistral-7B-Instruct-v0.3 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
          "python -m vllm.entrypoints.openai.api_server --served-model-name qwen3-235b-a22b --port 8995 --model /models/Qwen3-235B-A22B --tensor-parallel-size 8 --max-model-len 32000 --gpu-memory-utilization 0.80 --enable-reasoning --reasoning-parser deepseek_r1 --api-key <YOUR_API_KEY>"
        ]
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: 0,1,2,3,4,5,6,7
        ports:
        - containerPort: 8995
        resources:
          limits:
            volcano.sh/vgpu-number: '8'
        volumeMounts:
          - mountPath: /models/Qwen3-235B-A22B
            name: model-vol
          - mountPath: /dev/shm
            name: dshm
        livenessProbe:
          failureThreshold: 1
          httpGet:
            path: /health
            port: 8995
            scheme: HTTP
          initialDelaySeconds: 750
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8995
            scheme: HTTP
          initialDelaySeconds: 750
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5

vLLM 思考模式和非思考模式

vLLM qwen3moe 吞吐量较低

qwen3moe 在 2000 个 token 以上的提示下吞吐量较低

当token很少时，它运行正常，但随着输入的增加，它变得异常缓慢，就像Flash Attention之前的Transformers那样。在多GPU配置下，这个问题会更加严重，而单GPU的速度也比预期的要慢。

镜像: vllm/vllm-openai:v0.8.5.post1
硬件: 8xA100 80GB NVLink
CUDA 12.6
Prefill size is 20000~24000 tokens

使用 V0 引擎进行测试，强制执行开启和关闭，后端 FLASHINFER 和 FA2，专家并行开启和关闭，但所有组合都产生相同的结果。

Qwen3-30B-A3B/tp1

vllm serve /models/qwen/Qwen3-30B-A3B --port 7777 --tensor-parallel-size 1 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

21.0 tokens generated/sec on vllm

Qwen3-30B-A3B/tp8

vllm serve /models/qwen/Qwen3-30B-A3B --port 7777 --tensor-parallel-size 8 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

4.5 tokens generated/sec on vllm
80.0 tokens generated/sec on SGLang

Qwen3-235B-A22B/tp8

vllm serve /models/qwen/Qwen3-235B-A22B --port 7777 --tensor-parallel-size 8 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

2.2 tokens generated/sec on vllm
36.0 tokens generated/se on SGLang

SGLang 部署 Qwen3-235B-A22B

SGLang Qwen3-235B-A22B 启动

### server
# FA3 (default)
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080

# Flashinfer
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080 --attention-backend flashinfer

# Hybrid (FA3 for prefill and Flashinfer for decode)
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080 --enable-flashinfer-attention-decode

### client
python3 -m sglang.bench_serving --backend sglang \
            --dataset-name random \
            --dataset-path /data/datasets/ShareGPT_V3_unfiltered_cleaned_split.json \
            --random-input-len 3500 \
            --random-output-len 1500 \
            --random-range-ratio 1 \
            --request-rate 32 \
            --max-concurrency 32 \
            --num-prompts 128 \
            --host 0.0.0.0 --port 8080

SGLang 使用 Kubernetes 部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang
  labels:
    app: sglang
spec:
  selector:
    matchLabels:
      app: sglang
  replicas: 1
  template:
    metadata:
      labels:
        app: sglang
    spec:
      containers:
      - name: sglang
        image: lmsysorg/sglang:v0.4.6.post2-cu124
        command:
        - bash
        - -c
        - |
          set -x
          python3 -m sglang.launch_server \
            --host 0.0.0.0 \
            --port 50050 \
            --model-path /data/Qwen/Qwen3-235B-A22B \
            --served-model-name Qwen3-235B-A22B \
            --enable-torch-compile \
            --tp 8 \
            --reasoning-parser qwen3
        resources:
          limits:
            nvidia.com/gpu: "8"
        ports:
        - containerPort: 50050
        volumeMounts:
        - name: data
          mountPath: /data
        - name: shm
          mountPath: /dev/shm
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: models
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi
      restartPolicy: Always