vLLM 推理框架：Qwen3-235B-A22B 部署

vLLM 部署 Qwen3-235B-A22B-Instruct-2507

启动命令

sudo docker run --gpus '"device=0,1,2,3,4,5,6,7"'   -v /data/models/:/data/models/   -p 30001:30001    --ipc=host   --restart=always   -e TZ=UTC   harbor-cmp.zoomlion.com/library/vllm/vllm-openai:v0.10.1.1   --host 0.0.0.0 --port 30001   --max-model-len 131072   --tensor-parallel-size 8   --served-model-name Qwen3-235B-A22B-Instruct-2507   --model /data/models/Qwen3-235B-A22B-Instruct-2507   --enable-chunked-prefill   --enable-prefix-caching --gpu-memory-utilization 0.95    --rope-scaling '{"rope_type":"yarn", "factor":4.0,"original_max_position_embeddings":32768}'  --enable-auto-tool-choice   --tool-call-parser hermes   --trust-remote-code  --api-key df444f1463bd4fc3847fd6c50512f7b4

启动日志

• 日志

  INFO 08-27 08:58:50 [__init__.py:241] Automatically detected platform cuda.
  
  (APIServer pid=1) INFO 08-27 08:58:52 [api_server.py:1805] vLLM API server version 0.10.1.1
  
  (APIServer pid=1) INFO 08-27 08:58:52 [utils.py:326] non-default args: {'host': '0.0.0.0', 'port': 30001, 'api_key': ['df444f1463bd4fc3847fd6c50512f7b4'], 'enable_auto_tool_choice': True, 'tool_call_parser': 'hermes', 'model': '/data/models/Qwen3-235B-A22B-Instruct-2507', 'trust_remote_code': True, 'rope_scaling': {'rope_type': 'yarn', 'factor': 4.0, 'original_max_position_embeddings': 32768}, 'max_model_len': 131072, 'served_model_name': ['Qwen3-235B-A22B-Instruct-2507'], 'tensor_parallel_size': 8, 'gpu_memory_utilization': 0.95, 'enable_prefix_caching': True, 'enable_chunked_prefill': True}
  
  (APIServer pid=1) The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored.
  
  (APIServer pid=1) INFO 08-27 08:59:00 [__init__.py:711] Resolved architecture: Qwen3MoeForCausalLM
  
  (APIServer pid=1) INFO 08-27 08:59:00 [__init__.py:1750] Using max model len 131072
  
  (APIServer pid=1) INFO 08-27 08:59:01 [scheduler.py:222] Chunked prefill is enabled with max_num_batched_tokens=2048.
  
  INFO 08-27 08:59:06 [__init__.py:241] Automatically detected platform cuda.
  
  (EngineCore_0 pid=269) INFO 08-27 08:59:07 [core.py:636] Waiting for init message from front-end.
  
  (EngineCore_0 pid=269) INFO 08-27 08:59:07 [core.py:74] Initializing a V1 LLM engine (v0.10.1.1) with config: model='/data/models/Qwen3-235B-A22B-Instruct-2507', speculative_config=None, tokenizer='/data/models/Qwen3-235B-A22B-Instruct-2507', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=131072, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-235B-A22B-Instruct-2507, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":1,"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
  
  (EngineCore_0 pid=269) WARNING 08-27 08:59:07 [multiproc_worker_utils.py:273] Reducing Torch parallelism from 64 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
  
  (EngineCore_0 pid=269) INFO 08-27 08:59:07 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0, 1, 2, 3, 4, 5, 6, 7], buffer_handle=(8, 16777216, 10, 'psm_646322b2'), local_subscribe_addr='ipc:///tmp/945277c3-8187-42bf-ac05-1e19c75c72db', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  INFO 08-27 08:59:11 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  INFO 08-27 08:59:12 [__init__.py:241] Automatically detected platform cuda.
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_8a0077a3'), local_subscribe_addr='ipc:///tmp/12bc41f4-1b06-47b1-ab2f-a57623091d7b', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_4a7ed55e'), local_subscribe_addr='ipc:///tmp/f0d8295d-b015-4429-a5b6-86333c0165be', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_fc848ac2'), local_subscribe_addr='ipc:///tmp/fbe21211-a489-45c8-8446-1b919c205db1', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_36cb2e1f'), local_subscribe_addr='ipc:///tmp/a142c6de-1130-4932-a55d-e2df12f70536', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_ec3bb067'), local_subscribe_addr='ipc:///tmp/2c7b0708-77c9-45b3-9f78-35dc722a59aa', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_2c796bdc'), local_subscribe_addr='ipc:///tmp/7195b51b-59da-4d70-9b8c-ad117dcce680', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_539a5cd0'), local_subscribe_addr='ipc:///tmp/db1377c8-4376-4c71-a36b-bb0e8d102ca7', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:17 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_ec48d8c7'), local_subscribe_addr='ipc:///tmp/bebf4771-b29c-4321-aa5e-cf55fb16fe2a', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:20 [__init__.py:1418] Found nccl from library libnccl.so.2
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:20 [pynccl.py:70] vLLM is using nccl==2.26.2
  
  (VllmWorker TP7 pid=409) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP4 pid=406) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP0 pid=402) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP6 pid=408) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP2 pid=404) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP5 pid=407) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP1 pid=403) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP3 pid=405) WARNING 08-27 08:59:21 [custom_all_reduce.py:137] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [shm_broadcast.py:289] vLLM message queue communication handle: Handle(local_reader_ranks=[1, 2, 3, 4, 5, 6, 7], buffer_handle=(7, 4194304, 6, 'psm_65c03bcd'), local_subscribe_addr='ipc:///tmp/a4d9cec6-12e1-4890-859f-c0fdfc5353a5', remote_subscribe_addr=None, remote_addr_ipv6=False)
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 5 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 5, EP rank 5
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 0 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 3 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 3, EP rank 3
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 7 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 7, EP rank 7
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 6 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 6, EP rank 6
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 2 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 2, EP rank 2
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 4 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 4, EP rank 4
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:21 [parallel_state.py:1134] rank 1 in world size 8 is assigned as DP rank 0, PP rank 0, TP rank 1, EP rank 1
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:21 [topk_topp_sampler.py:50] Using FlashInfer for top-p & top-k sampling.
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:21 [gpu_model_runner.py:1953] Starting to load model /data/models/Qwen3-235B-A22B-Instruct-2507...
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP4 pid=406) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP5 pid=407) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [gpu_model_runner.py:1985] Loading model from scratch...
  
  (VllmWorker TP2 pid=404) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP3 pid=405) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP7 pid=409) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP6 pid=408) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP1 pid=403) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  (VllmWorker TP0 pid=402) INFO 08-27 08:59:21 [cuda.py:328] Using Flash Attention backend on V1 engine.
  
  Loading safetensors checkpoint shards:   0% Completed | 0/118 [00:00<?, ?it/s]
  
  Loading safetensors checkpoint shards:   1% Completed | 1/118 [00:01<02:34,  1.32s/it]
  
  Loading safetensors checkpoint shards:   2% Completed | 2/118 [00:02<02:30,  1.29s/it]
  
  Loading safetensors checkpoint shards:   3% Completed | 3/118 [00:02<01:36,  1.19it/s]
  
  Loading safetensors checkpoint shards:   3% Completed | 4/118 [00:03<01:09,  1.65it/s]
  
  Loading safetensors checkpoint shards:   4% Completed | 5/118 [00:03<01:13,  1.55it/s]
  
  Loading safetensors checkpoint shards:   5% Completed | 6/118 [00:04<00:56,  1.97it/s]
  
  Loading safetensors checkpoint shards:   6% Completed | 7/118 [00:04<00:46,  2.37it/s]
  
  Loading safetensors checkpoint shards:   7% Completed | 8/118 [00:04<00:39,  2.75it/s]
  
  Loading safetensors checkpoint shards:   8% Completed | 9/118 [00:04<00:36,  2.97it/s]
  
  Loading safetensors checkpoint shards:   8% Completed | 10/118 [00:05<00:34,  3.11it/s]
  
  Loading safetensors checkpoint shards:   9% Completed | 11/118 [00:05<00:33,  3.24it/s]
  
  Loading safetensors checkpoint shards:  10% Completed | 12/118 [00:05<00:30,  3.43it/s]
  
  Loading safetensors checkpoint shards:  11% Completed | 13/118 [00:05<00:29,  3.56it/s]
  
  Loading safetensors checkpoint shards:  12% Completed | 14/118 [00:06<00:28,  3.71it/s]
  
  Loading safetensors checkpoint shards:  13% Completed | 15/118 [00:06<00:27,  3.80it/s]
  
  Loading safetensors checkpoint shards:  14% Completed | 16/118 [00:09<01:50,  1.09s/it]
  
  Loading safetensors checkpoint shards:  14% Completed | 17/118 [00:10<02:04,  1.23s/it]
  
  Loading safetensors checkpoint shards:  15% Completed | 18/118 [00:11<01:40,  1.01s/it]
  
  Loading safetensors checkpoint shards:  16% Completed | 19/118 [00:12<01:27,  1.14it/s]
  
  Loading safetensors checkpoint shards:  17% Completed | 20/118 [00:12<01:08,  1.44it/s]
  
  Loading safetensors checkpoint shards:  18% Completed | 21/118 [00:12<01:05,  1.49it/s]
  
  Loading safetensors checkpoint shards:  19% Completed | 22/118 [00:13<01:02,  1.55it/s]
  
  Loading safetensors checkpoint shards:  19% Completed | 23/118 [00:13<00:55,  1.71it/s]
  
  Loading safetensors checkpoint shards:  20% Completed | 24/118 [00:14<00:54,  1.73it/s]
  
  Loading safetensors checkpoint shards:  21% Completed | 25/118 [00:15<00:52,  1.77it/s]
  
  Loading safetensors checkpoint shards:  22% Completed | 26/118 [00:15<00:51,  1.80it/s]
  
  Loading safetensors checkpoint shards:  23% Completed | 27/118 [00:15<00:41,  2.19it/s]
  
  Loading safetensors checkpoint shards:  24% Completed | 28/118 [00:16<00:42,  2.13it/s]
  
  Loading safetensors checkpoint shards:  25% Completed | 29/118 [00:16<00:42,  2.08it/s]
  
  Loading safetensors checkpoint shards:  25% Completed | 30/118 [00:17<00:43,  2.04it/s]
  
  Loading safetensors checkpoint shards:  26% Completed | 31/118 [00:17<00:43,  2.01it/s]
  
  Loading safetensors checkpoint shards:  27% Completed | 32/118 [00:18<00:36,  2.36it/s]
  
  Loading safetensors checkpoint shards:  28% Completed | 33/118 [00:18<00:33,  2.57it/s]
  
  Loading safetensors checkpoint shards:  29% Completed | 34/118 [00:18<00:35,  2.35it/s]
  
  Loading safetensors checkpoint shards:  30% Completed | 35/118 [00:19<00:37,  2.21it/s]
  
  Loading safetensors checkpoint shards:  31% Completed | 36/118 [00:19<00:37,  2.17it/s]
  
  Loading safetensors checkpoint shards:  31% Completed | 37/118 [00:20<00:38,  2.12it/s]
  
  Loading safetensors checkpoint shards:  32% Completed | 38/118 [00:20<00:38,  2.07it/s]
  
  Loading safetensors checkpoint shards:  33% Completed | 39/118 [00:21<00:38,  2.04it/s]
  
  Loading safetensors checkpoint shards:  34% Completed | 40/118 [00:21<00:38,  2.04it/s]
  
  Loading safetensors checkpoint shards:  35% Completed | 41/118 [00:22<00:34,  2.23it/s]
  
  Loading safetensors checkpoint shards:  36% Completed | 42/118 [00:22<00:35,  2.15it/s]
  
  Loading safetensors checkpoint shards:  36% Completed | 43/118 [00:23<00:35,  2.10it/s]
  
  Loading safetensors checkpoint shards:  37% Completed | 44/118 [00:23<00:33,  2.18it/s]
  
  Loading safetensors checkpoint shards:  38% Completed | 45/118 [00:24<00:34,  2.11it/s]
  
  Loading safetensors checkpoint shards:  39% Completed | 46/118 [00:24<00:31,  2.28it/s]
  
  Loading safetensors checkpoint shards:  40% Completed | 47/118 [00:25<00:31,  2.23it/s]
  
  Loading safetensors checkpoint shards:  41% Completed | 48/118 [00:25<00:30,  2.32it/s]
  
  Loading safetensors checkpoint shards:  42% Completed | 49/118 [00:25<00:29,  2.36it/s]
  
  Loading safetensors checkpoint shards:  42% Completed | 50/118 [00:26<00:26,  2.56it/s]
  
  Loading safetensors checkpoint shards:  43% Completed | 51/118 [00:26<00:22,  2.95it/s]
  
  Loading safetensors checkpoint shards:  44% Completed | 52/118 [00:26<00:20,  3.24it/s]
  
  Loading safetensors checkpoint shards:  45% Completed | 53/118 [00:26<00:18,  3.52it/s]
  
  Loading safetensors checkpoint shards:  46% Completed | 54/118 [00:27<00:17,  3.68it/s]
  
  Loading safetensors checkpoint shards:  47% Completed | 55/118 [00:27<00:17,  3.56it/s]
  
  Loading safetensors checkpoint shards:  47% Completed | 56/118 [00:27<00:17,  3.53it/s]
  
  Loading safetensors checkpoint shards:  48% Completed | 57/118 [00:27<00:17,  3.43it/s]
  
  Loading safetensors checkpoint shards:  49% Completed | 58/118 [00:28<00:15,  3.87it/s]
  
  Loading safetensors checkpoint shards:  50% Completed | 59/118 [00:28<00:14,  4.09it/s]
  
  Loading safetensors checkpoint shards:  51% Completed | 60/118 [00:28<00:15,  3.83it/s]
  
  Loading safetensors checkpoint shards:  52% Completed | 61/118 [00:28<00:14,  4.00it/s]
  
  Loading safetensors checkpoint shards:  53% Completed | 62/118 [00:29<00:13,  4.13it/s]
  
  Loading safetensors checkpoint shards:  53% Completed | 63/118 [00:29<00:12,  4.26it/s]
  
  Loading safetensors checkpoint shards:  54% Completed | 64/118 [00:29<00:12,  4.32it/s]
  
  Loading safetensors checkpoint shards:  55% Completed | 65/118 [00:29<00:12,  4.27it/s]
  
  Loading safetensors checkpoint shards:  56% Completed | 66/118 [00:30<00:12,  4.24it/s]
  
  Loading safetensors checkpoint shards:  57% Completed | 67/118 [00:30<00:12,  4.20it/s]
  
  Loading safetensors checkpoint shards:  58% Completed | 68/118 [00:30<00:11,  4.26it/s]
  
  Loading safetensors checkpoint shards:  58% Completed | 69/118 [00:30<00:11,  4.30it/s]
  
  Loading safetensors checkpoint shards:  59% Completed | 70/118 [00:30<00:10,  4.45it/s]
  
  Loading safetensors checkpoint shards:  60% Completed | 71/118 [00:31<00:10,  4.37it/s]
  
  Loading safetensors checkpoint shards:  61% Completed | 72/118 [00:31<00:10,  4.36it/s]
  
  Loading safetensors checkpoint shards:  62% Completed | 73/118 [00:31<00:10,  4.32it/s]
  
  Loading safetensors checkpoint shards:  63% Completed | 74/118 [00:31<00:10,  4.36it/s]
  
  Loading safetensors checkpoint shards:  64% Completed | 75/118 [00:32<00:09,  4.39it/s]
  
  Loading safetensors checkpoint shards:  64% Completed | 76/118 [00:32<00:09,  4.31it/s]
  
  Loading safetensors checkpoint shards:  65% Completed | 77/118 [00:32<00:09,  4.27it/s]
  
  Loading safetensors checkpoint shards:  66% Completed | 78/118 [00:32<00:09,  4.32it/s]
  
  Loading safetensors checkpoint shards:  67% Completed | 79/118 [00:33<00:08,  4.35it/s]
  
  Loading safetensors checkpoint shards:  68% Completed | 80/118 [00:33<00:08,  4.39it/s]
  
  Loading safetensors checkpoint shards:  69% Completed | 81/118 [00:33<00:08,  4.50it/s]
  
  Loading safetensors checkpoint shards:  69% Completed | 82/118 [00:33<00:07,  4.57it/s]
  
  Loading safetensors checkpoint shards:  70% Completed | 83/118 [00:33<00:07,  4.60it/s]
  
  Loading safetensors checkpoint shards:  71% Completed | 84/118 [00:34<00:07,  4.51it/s]
  
  Loading safetensors checkpoint shards:  72% Completed | 85/118 [00:34<00:07,  4.45it/s]
  
  Loading safetensors checkpoint shards:  73% Completed | 86/118 [00:34<00:07,  4.42it/s]
  
  Loading safetensors checkpoint shards:  74% Completed | 87/118 [00:34<00:07,  4.38it/s]
  
  Loading safetensors checkpoint shards:  75% Completed | 88/118 [00:35<00:06,  4.38it/s]
  
  Loading safetensors checkpoint shards:  75% Completed | 89/118 [00:35<00:06,  4.42it/s]
  
  Loading safetensors checkpoint shards:  76% Completed | 90/118 [00:35<00:06,  4.46it/s]
  
  Loading safetensors checkpoint shards:  77% Completed | 91/118 [00:35<00:06,  4.47it/s]
  
  Loading safetensors checkpoint shards:  78% Completed | 92/118 [00:35<00:05,  4.54it/s]
  
  Loading safetensors checkpoint shards:  79% Completed | 93/118 [00:36<00:05,  4.62it/s]
  
  Loading safetensors checkpoint shards:  80% Completed | 94/118 [00:36<00:05,  4.62it/s]
  
  Loading safetensors checkpoint shards:  81% Completed | 95/118 [00:36<00:05,  4.58it/s]
  
  Loading safetensors checkpoint shards:  81% Completed | 96/118 [00:36<00:04,  4.55it/s]
  
  Loading safetensors checkpoint shards:  82% Completed | 97/118 [00:37<00:04,  4.56it/s]
  
  Loading safetensors checkpoint shards:  83% Completed | 98/118 [00:37<00:04,  4.59it/s]
  
  Loading safetensors checkpoint shards:  84% Completed | 99/118 [00:37<00:04,  4.65it/s]
  
  Loading safetensors checkpoint shards:  85% Completed | 100/118 [00:37<00:04,  4.28it/s]
  
  Loading safetensors checkpoint shards:  86% Completed | 101/118 [00:37<00:03,  4.42it/s]
  
  Loading safetensors checkpoint shards:  86% Completed | 102/118 [00:38<00:03,  4.16it/s]
  
  Loading safetensors checkpoint shards:  87% Completed | 103/118 [00:38<00:03,  4.33it/s]
  
  Loading safetensors checkpoint shards:  88% Completed | 104/118 [00:38<00:03,  4.45it/s]
  
  Loading safetensors checkpoint shards:  89% Completed | 105/118 [00:38<00:02,  4.49it/s]
  
  Loading safetensors checkpoint shards:  90% Completed | 106/118 [00:39<00:02,  4.50it/s]
  
  Loading safetensors checkpoint shards:  91% Completed | 107/118 [00:39<00:02,  4.55it/s]
  
  Loading safetensors checkpoint shards:  92% Completed | 108/118 [00:39<00:02,  4.54it/s]
  
  Loading safetensors checkpoint shards:  92% Completed | 109/118 [00:39<00:02,  4.40it/s]
  
  Loading safetensors checkpoint shards:  93% Completed | 110/118 [00:39<00:01,  4.19it/s]
  
  Loading safetensors checkpoint shards:  94% Completed | 111/118 [00:40<00:01,  4.10it/s]
  
  Loading safetensors checkpoint shards:  95% Completed | 112/118 [00:40<00:01,  4.06it/s]
  
  Loading safetensors checkpoint shards:  96% Completed | 113/118 [00:40<00:01,  4.08it/s]
  
  Loading safetensors checkpoint shards:  97% Completed | 114/118 [00:40<00:00,  4.15it/s]
  
  Loading safetensors checkpoint shards:  97% Completed | 115/118 [00:41<00:00,  4.18it/s]
  
  Loading safetensors checkpoint shards:  98% Completed | 116/118 [00:41<00:00,  4.16it/s]
  
  Loading safetensors checkpoint shards:  99% Completed | 117/118 [00:41<00:00,  4.23it/s]
  
  Loading safetensors checkpoint shards: 100% Completed | 118/118 [00:41<00:00,  4.15it/s]
  
  Loading safetensors checkpoint shards: 100% Completed | 118/118 [00:41<00:00,  2.81it/s]
  
  (VllmWorker TP0 pid=402)
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.05 seconds
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.07 seconds
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.07 seconds
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.07 seconds
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.05 seconds
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 42.02 seconds
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:00:04 [default_loader.py:262] Loading weights took 41.98 seconds
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.566111 seconds
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.552377 seconds
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.606039 seconds
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.613413 seconds
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.513302 seconds
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.601378 seconds
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:00:04 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 42.617622 seconds
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:00:05 [default_loader.py:262] Loading weights took 42.98 seconds
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:00:05 [gpu_model_runner.py:2007] Model loading took 54.9420 GiB and 43.553418 seconds
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:00:27 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_1_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:00:27 [backends.py:559] Dynamo bytecode transform time: 21.43 s
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:00:27 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_4_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:00:27 [backends.py:559] Dynamo bytecode transform time: 21.83 s
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:00:27 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_6_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:00:27 [backends.py:559] Dynamo bytecode transform time: 21.84 s
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:00:27 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_5_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:00:27 [backends.py:559] Dynamo bytecode transform time: 22.09 s
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:00:28 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_2_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:00:28 [backends.py:559] Dynamo bytecode transform time: 22.19 s
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:00:28 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_7_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:00:28 [backends.py:559] Dynamo bytecode transform time: 22.25 s
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:00:28 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_0_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:00:28 [backends.py:559] Dynamo bytecode transform time: 22.26 s
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:00:28 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/fa6ca574eb/rank_3_0/backbone for vLLM's torch.compile
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:00:28 [backends.py:559] Dynamo bytecode transform time: 22.53 s
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:00:34 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:00:34 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:00:35 [backends.py:194] Cache the graph for dynamic shape for later use
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:02:29 [backends.py:215] Compiling a graph for dynamic shape takes 121.13 s
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:02:30 [backends.py:215] Compiling a graph for dynamic shape takes 121.73 s
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:02:30 [backends.py:215] Compiling a graph for dynamic shape takes 122.00 s
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:02:31 [backends.py:215] Compiling a graph for dynamic shape takes 122.99 s
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:02:31 [backends.py:215] Compiling a graph for dynamic shape takes 123.34 s
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:02:32 [backends.py:215] Compiling a graph for dynamic shape takes 123.16 s
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:02:32 [backends.py:215] Compiling a graph for dynamic shape takes 123.42 s
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:02:32 [backends.py:215] Compiling a graph for dynamic shape takes 123.15 s
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:02:36 [fused_moe.py:720] Using configuration from /usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/fused_moe/configs/E=128,N=192,device_name=NVIDIA_A100-SXM4-80GB.json for MoE layer.
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 142.55 s in total
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 143.91 s in total
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 144.09 s in total
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 144.81 s in total
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 145.40 s in total
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 145.19 s in total
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 145.68 s in total
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:02:48 [monitor.py:34] torch.compile takes 145.68 s in total
  
  (VllmWorker TP4 pid=406) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP4 pid=406) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP4 pid=406)   warnings.warn(
  
  (VllmWorker TP3 pid=405) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP3 pid=405) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP3 pid=405)   warnings.warn(
  
  (VllmWorker TP7 pid=409) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP7 pid=409) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP7 pid=409)   warnings.warn(
  
  (VllmWorker TP5 pid=407) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP5 pid=407) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP5 pid=407)   warnings.warn(
  
  (VllmWorker TP6 pid=408) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP6 pid=408) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP6 pid=408)   warnings.warn(
  
  (VllmWorker TP2 pid=404) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP2 pid=404) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP2 pid=404)   warnings.warn(
  
  (VllmWorker TP0 pid=402) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP0 pid=402) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP0 pid=402)   warnings.warn(
  
  (VllmWorker TP1 pid=403) /usr/local/lib/python3.12/dist-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
  
  (VllmWorker TP1 pid=403) If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
  
  (VllmWorker TP1 pid=403)   warnings.warn(
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.51 GiB
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.51 GiB
  
  (VllmWorker TP1 pid=403) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.47 GiB
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.51 GiB
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.47 GiB
  
  (VllmWorker TP5 pid=407) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.47 GiB
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.51 GiB
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:03:41 [gpu_worker.py:276] Available KV cache memory: 19.47 GiB
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 435,344 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.32x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 434,288 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.31x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 434,288 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.31x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 435,344 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.32x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 435,344 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.32x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 434,288 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.31x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 434,288 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.31x
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:849] GPU KV cache size: 435,344 tokens
  
  (EngineCore_0 pid=269) INFO 08-27 09:03:42 [kv_cache_utils.py:853] Maximum concurrency for 131,072 tokens per request: 3.32x
  
  Capturing CUDA graphs (mixed prefill-decode, PIECEWISE):  99%|█████████▊| 66/67 00:28<00:00,  1.13it/s INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (VllmWorker TP6 pid=408) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (VllmWorker TP2 pid=404) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (VllmWorker TP3 pid=405) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 00:29<00:00,  1.06it/s INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:29<00:00,  2.27it/s]
  
  (VllmWorker TP4 pid=406) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (VllmWorker TP0 pid=402) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (VllmWorker TP7 pid=409) INFO 08-27 09:04:13 [gpu_model_runner.py:2708] Graph capturing finished in 31 secs, took 2.56 GiB
  
  (EngineCore_0 pid=269) INFO 08-27 09:04:13 [core.py:214] init engine (profile, create kv cache, warmup model) took 248.22 seconds
  
  (APIServer pid=1) INFO 08-27 09:04:14 [loggers.py:142] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 27143
  
  (APIServer pid=1) INFO 08-27 09:04:14 [api_server.py:1611] Supported_tasks: ['generate']
  
  (APIServer pid=1) WARNING 08-27 09:04:14 [__init__.py:1625] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
  
  (APIServer pid=1) INFO 08-27 09:04:14 [serving_responses.py:120] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
  
  (APIServer pid=1) INFO 08-27 09:04:14 [serving_responses.py:149] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
  
  (APIServer pid=1) INFO 08-27 09:04:14 [serving_chat.py:94] "auto" tool choice has been enabled please note that while the parallel_tool_calls client option is preset for compatibility reasons, it will be ignored.
  
  (APIServer pid=1) INFO 08-27 09:04:14 [serving_chat.py:134] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
  
  (APIServer pid=1) INFO 08-27 09:04:14 [serving_completion.py:77] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
  
  (APIServer pid=1) INFO 08-27 09:04:14 [api_server.py:1880] Starting vLLM API server 0 on http://0.0.0.0:30001
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:36] Available routes are:
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /openapi.json, Methods: HEAD, GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /docs, Methods: HEAD, GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /docs/oauth2-redirect, Methods: HEAD, GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /redoc, Methods: HEAD, GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /health, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /load, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /ping, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /ping, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /tokenize, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /detokenize, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/models, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /version, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/responses, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/responses/{response_id}, Methods: GET
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/responses/{response_id}/cancel, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/chat/completions, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/completions, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/embeddings, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /pooling, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /classify, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /score, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/score, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/audio/transcriptions, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/audio/translations, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /rerank, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v1/rerank, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /v2/rerank, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /scale_elastic_ep, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /is_scaling_elastic_ep, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /invocations, Methods: POST
  
  (APIServer pid=1) INFO 08-27 09:04:14 [launcher.py:44] Route: /metrics, Methods: GET
  
  (APIServer pid=1) INFO:     Started server process [1]
  
  (APIServer pid=1) INFO:     Waiting for application startup.
  
  (APIServer pid=1) INFO:     Application startup complete.

调用日志

(APIServer pid=1) INFO:     10.33.27.93:55840 - "POST /v1/chat/completions HTTP/1.1" 200 OK

(APIServer pid=1) INFO 08-27 09:04:24 [loggers.py:123] Engine 000: Avg prompt throughput: 2.3 tokens/s, Avg generation throughput: 7.9 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%

vLLM 部署 Qwen3-235B-A22B

https://github.com/vllm-project/vllm/issues/17327

vLLM Qwen3-235B-A22B 启动

20250430版本支持qwen3思考/非思考模式

1、–enable-prefix-caching、–chunked-prefill-enabled、–use-v2-block-manager默认开启；

2、–enforce-eager、–enable-expert-parallel默认关闭；

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server \
    --served-model-name qwen3-235b-a22b \
    --port 8995 \
    --model /models/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --max-model-len 32000 \
    --gpu-memory-utilization 0.80 \
    --enable-reasoning \
    --reasoning-parser deepseek_r1 \
    --api-key 669b12de160848509c3a0ba5d7704729

vLLM Qwen3-235B-A22B 优化

1、–enable-expert-parallel开启专家并行，Qwen3-235B-A22B-FP8的–tensor-parallel-size为4；

2、–enable-reasoning –reasoning-parser deepseek_r1 –enable-auto-tool-choice –tool-call-parser hermes来支持MCP；

3、–max-num-batched-tokens 、–max_num_seqs默认为None；

CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 python -m vllm.entrypoints.openai.api_server \
    --served-model-name qwen3-235b-a22b \
    --port 8995 \
    --model /models/Qwen3-235B-A22B \
    --tensor-parallel-size 8 \
    --enable-expert-parallel \
    --max-model-len 32000 \
    --gpu-memory-utilization 0.80 \
    --enable-reasoning \
    --reasoning-parser deepseek_r1 \
    --enable-auto-tool-choice \
    --tool-call-parser hermes \
    --api-key 669b12de160848509c3a0ba5d7704729

–cpu-offload-gb：将每张卡应该装载到显存的模型卸载到CPU内存中，这个性能受限于PCIe速度和内存带宽，卸载到内存中的模型每次前向传播都会装载到GPU的显存中。

vLLM 0.8.5日志分析

• vLLM API server version 0.8.5

INFO 04-29 01:18:54 [api_server.py:1044] args: Namespace(host=None, port=8995, uvicorn_log_level='info', disable_uvicorn_access_log=False, allow_credentials=False, allowed_origins=[''], allowed_methods=[''], allowed_headers=['*'], api_key='669b12de160848509c3a0ba5d7704729', lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, enable_ssl_refresh=False, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='/models/Qwen3-235B-A22B', task='auto', tokenizer=None, hf_config_path=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, load_format='auto', download_dir=None, model_loader_extra_config={}, use_tqdm_on_load=True, config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', max_model_len=32000, guided_decoding_backend='auto', reasoning_parser='deepseek_r1', logits_processor_pattern=None, model_impl='auto', distributed_executor_backend=None, pipeline_parallel_size=1, tensor_parallel_size=8, data_parallel_size=1, enable_expert_parallel=False, max_parallel_loading_workers=None, ray_workers_use_nsight=False, disable_custom_all_reduce=False, block_size=None, gpu_memory_utilization=0.9, swap_space=4, kv_cache_dtype='auto', num_gpu_blocks_override=None, enable_prefix_caching=None, prefix_caching_hash_algo='builtin', cpu_offload_gb=0, calculate_kv_scales=False, disable_sliding_window=False, use_v2_block_manager=True, seed=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_token=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config={}, limit_mm_per_prompt={}, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=None, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=None, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', speculative_config=None, ignore_patterns=[], served_model_name=['qwen3-235b-a22b'], qlora_adapter_name_or_path=None, show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, max_num_batched_tokens=None, max_num_seqs=None, max_num_partial_prefills=1, max_long_partial_prefills=1, long_prefill_token_threshold=0, num_lookahead_slots=0, scheduler_delay_factor=0.0, preemption_mode=None, num_scheduler_steps=1, multi_step_stream_outputs=True, scheduling_policy='fcfs', enable_chunked_prefill=None, disable_chunked_mm_input=False, scheduler_cls='vllm.core.scheduler.Scheduler', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', worker_extension_cls='', generation_config='auto', override_generation_config=None, enable_sleep_mode=False, additional_config=None, enable_reasoning=True, disable_cascade_attn=False, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, enable_server_load_tracking=False)

• This model supports multiple tasks: {‘embed’, ‘classify’, ‘generate’, ‘reward’, ‘score’}. Defaulting to ‘generate’.
• Defaulting to use mp for distributed inference
• Chunked prefill is enabled with max_num_batched_tokens=8192.

INFO 04-29 01:19:11 [core.py:58] Initializing a V1 LLM engine (v0.8.5) with config: model='/models/Qwen3-235B-A22B', speculative_config=None, tokenizer='/models/Qwen3-235B-A22B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32000, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend='deepseek_r1'), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=qwen3-235b-a22b, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}

• vLLM is using nccl==2.21.5
• Using Flash Attention backend on V1 engine.
• Using FlashInfer for top-p & top-k sampling.
• Using default completion sampling params from model: {‘temperature’: 0.6, ‘top_k’: 20, ‘top_p’: 0.95}

vLLM 0.8.5调用日志

INFO 05-18 20:11:58 [logger.py:39] Received request chatcmpl-ba69850c020242479e52db988c7834ca: prompt: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n你是谁？<|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.6, top_p=0.95, top_k=20, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=31978, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None), prompt_token_ids: None, lora_request: None, prompt_adapter_request: None.

思考模式：流式返回的首个token

{
    "id": "chatcmpl-ba69850c020242479e52db988c7834ca",
    "object": "chat.completion.chunk",
    "created": 1747624318,
    "model": "qwen3-235b-a22b",
    "choices": [
        {
            "index": 0,
            "delta": {
                "role": "assistant",
                "content": ""
            },
            "logprobs": null,
            "finish_reason": null
        }
    ]
}

思考模式：非流式返回

{
    "id": "chatcmpl-c85543dd23024d99944f76ff2e511120",
    "object": "chat.completion",
    "created": 1747624579,
    "model": "qwen3-235b-a22b",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "reasoning_content": "\n\n好的，用户问“你是谁？”。。。\n",
                "content": "\n\n你好！我是通义千问。。。。",
                "tool_calls": []
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null
        }
    ],
    "usage": {
        "prompt_tokens": 22,
        "total_tokens": 324,
        "completion_tokens": 302,
        "prompt_tokens_details": null
    },
    "prompt_logprobs": null

vLLM 使用 Kubernetes 部署

Using Kubernetes — vLLM

apiVersion: apps/v1
kind: Deployment
metadata:
  name: qwen3-235b
  namespace: custom-model-deploy
  labels:
    app: qwen3-235b
spec:
  replicas: 1
  selector:
    matchLabels:
      app: qwen3-235b
  template:
    metadata:
      labels:
        app: qwen3-235b
    spec:
      volumes:
      # vLLM needs to access the host's shared memory for tensor parallel inference.
      - name: dshm
        emptyDir:
          medium: Memory
          sizeLimit: "8Gi"
        hostPath:
          path: /data/models/Qwen3-235B-A22B
          type: ''
        name: model-vol
      containers:
      - name: qwen3-235b
        image: vllm/vllm-openai:v0.8.5.post1
        command: ["/bin/sh", "-c"]
        args: [
          "python3 serve mistralai/Mistral-7B-Instruct-v0.3 --trust-remote-code --enable-chunked-prefill --max_num_batched_tokens 1024"
          "python -m vllm.entrypoints.openai.api_server --served-model-name qwen3-235b-a22b --port 8995 --model /models/Qwen3-235B-A22B --tensor-parallel-size 8 --max-model-len 32000 --gpu-memory-utilization 0.80 --enable-reasoning --reasoning-parser deepseek_r1 --api-key 669b12de160848509c3a0ba5d7704729"
        ]
        env:
        - name: CUDA_VISIBLE_DEVICES
          value: 0,1,2,3,4,5,6,7
        ports:
        - containerPort: 8995
        resources:
          limits:
            volcano.sh/vgpu-number: '8'
        volumeMounts:
          - mountPath: /models/Qwen3-235B-A22B
            name: model-vol
          - mountPath: /dev/shm
            name: dshm
        livenessProbe:
          failureThreshold: 1
          httpGet:
            path: /health
            port: 8995
            scheme: HTTP
          initialDelaySeconds: 750
          periodSeconds: 30
          successThreshold: 1
          timeoutSeconds: 10
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /health
            port: 8995
            scheme: HTTP
          initialDelaySeconds: 750
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 5

vLLM 思考模式和非思考模式

!企业微信截图_17470205289598.png

vLLM qwen3moe 吞吐量较低

https://github.com/vllm-project/vllm/issues/17650

qwen3moe 在 2000 个 token 以上的提示下吞吐量较低

当token很少时，它运行正常，但随着输入的增加，它变得异常缓慢，就像Flash Attention之前的Transformers那样。在多GPU配置下，这个问题会更加严重，而单GPU的速度也比预期的要慢。

镜像: vllm/vllm-openai:v0.8.5.post1

硬件: 8xA100 80GB NVLink

CUDA 12.6

Prefill size is 20000~24000 tokens

使用 V0 引擎进行测试，强制执行开启和关闭，后端 FLASHINFER 和 FA2，专家并行开启和关闭，但所有组合都产生相同的结果。

Qwen3-30B-A3B/tp1

vllm serve /models/qwen/Qwen3-30B-A3B --port 7777 --tensor-parallel-size 1 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

21.0 tokens generated/sec on vllm

Qwen3-30B-A3B/tp8

vllm serve /models/qwen/Qwen3-30B-A3B --port 7777 --tensor-parallel-size 8 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

4.5 tokens generated/sec on vllm

80.0 tokens generated/sec on SGLang

Qwen3-235B-A22B/tp8

vllm serve /models/qwen/Qwen3-235B-A22B --port 7777 --tensor-parallel-size 8 --max-num-batched-tokens 131072 --max-model-len 32768 --enable-prefix-caching

2.2 tokens generated/sec on vllm

36.0 tokens generated/se on SGLang

SGLang 部署 Qwen3-235B-A22B

https://github.com/sgl-project/sglang/issues/6066

https://github.com/sgl-project/sglang/pull/6151

SGLang Qwen3-235B-A22B 启动

### server
# FA3 (default)
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080

# Flashinfer
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080 --attention-backend flashinfer

# Hybrid (FA3 for prefill and Flashinfer for decode)
python3 -m sglang.launch_server --model /data/models/Qwen3-235B-A22B --tp 8 --reasoning-parser qwen3 --port 8080 --enable-flashinfer-attention-decode

### client
python3 -m sglang.bench_serving --backend sglang \
            --dataset-name random \
            --dataset-path /data/datasets/ShareGPT_V3_unfiltered_cleaned_split.json \
            --random-input-len 3500 \
            --random-output-len 1500 \
            --random-range-ratio 1 \
            --request-rate 32 \
            --max-concurrency 32 \
            --num-prompts 128 \
            --host 0.0.0.0 --port 8080

SGLang 使用 Kubernetes 部署

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sglang
  labels:
    app: sglang
spec:
  selector:
    matchLabels:
      app: sglang
  replicas: 1
  template:
    metadata:
      labels:
        app: sglang
    spec:
      containers:
      - name: sglang
        image: lmsysorg/sglang:v0.4.6.post2-cu124
        command:
        - bash
        - -c
        - |
          set -x
          python3 -m sglang.launch_server \
            --host 0.0.0.0 \
            --port 50050 \
            --model-path /data/Qwen/Qwen3-235B-A22B \
            --served-model-name Qwen3-235B-A22B \
            --enable-torch-compile \
            --tp 8 \
            --reasoning-parser qwen3
        resources:
          limits:
            nvidia.com/gpu: "8"
        ports:
        - containerPort: 50050
        volumeMounts:
        - name: data
          mountPath: /data
        - name: shm
          mountPath: /dev/shm
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: models
      - name: shm
        emptyDir:
          medium: Memory
          sizeLimit: 64Gi
      restartPolicy: Always