vLLM 0.11.0部署Qwen3-VL-2B-Instruct实战：Docker启动、OpenAI兼容API调用与返回参数解析

发表于 | 分类于 | 2k | 12 分钟

vLLM 推理框架—Qwen3-VL-2B-Instruct部署

1、启动命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
sudo docker run --gpus '"device=2"' \
  -v /data/models/:/data/models/ \
  -p 30001:30001 \
  --ipc=host \
  --restart=always \
  -e TZ=UTC \
  harbor-cmp.zoomlion.com/library/vllm/vllm-openai:v0.11.0 \
  --model /data/models/Qwen3-VL-2B-Instruct \
  --served-model-name Qwen3-VL-2B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.95 \
  --disable-log-stats \
  --limit_mm_per_prompt.image 10  \
  --api-key <YOUR_API_KEY> \
  --host 0.0.0.0 \
  --port 30001

2、启动日志

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
INFO 10-22 06:02:17 [__init__.py:216] Automatically detected platform cuda.
(APIServer pid=1) INFO 10-22 06:02:22 [api_server.py:1839] vLLM API server version 0.11.0
(APIServer pid=1) INFO 10-22 06:02:22 [utils.py:233] non-default args: {'host': '0.0.0.0', 'port': 30001, 'api_key': ['51b41204620745338ff7b01cc1151a2a'], 'model': '/data/models/Qwen3-VL-2B-Instruct', 'max_model_len': 32768, 'served_model_name': ['Qwen3-VL-2B-Instruct'], 'gpu_memory_utilization': 0.95, 'limit_mm_per_prompt': {'image': 10}, 'disable_log_stats': True}
(APIServer pid=1) INFO 10-22 06:02:34 [model.py:547] Resolved architecture: Qwen3VLForConditionalGeneration
(APIServer pid=1) `torch_dtype` is deprecated! Use `dtype` instead!
(APIServer pid=1) INFO 10-22 06:02:34 [model.py:1510] Using max model len 32768
(APIServer pid=1) INFO 10-22 06:02:35 [scheduler.py:205] Chunked prefill is enabled with max_num_batched_tokens=2048.
INFO 10-22 06:02:40 [__init__.py:216] Automatically detected platform cuda.
(EngineCore_DP0 pid=269) INFO 10-22 06:02:42 [core.py:644] Waiting for init message from front-end.
(EngineCore_DP0 pid=269) INFO 10-22 06:02:42 [core.py:77] Initializing a V1 LLM engine (v0.11.0) with config: model='/data/models/Qwen3-VL-2B-Instruct', speculative_config=None, tokenizer='/data/models/Qwen3-VL-2B-Instruct', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=Qwen3-VL-2B-Instruct, enable_prefix_caching=True, chunked_prefill_enabled=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output","vllm.mamba_mixer2","vllm.mamba_mixer","vllm.short_conv","vllm.linear_attention","vllm.plamo2_mamba_mixer","vllm.gdn_attention","vllm.sparse_attn_indexer"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"cudagraph_mode":[2,1],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"use_inductor_graph_partition":false,"pass_config":{},"max_capture_size":512,"local_cache_dir":null}
(EngineCore_DP0 pid=269) W1022 06:02:45.306000 269 torch/utils/cpp_extension.py:2425] TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation. 
(EngineCore_DP0 pid=269) W1022 06:02:45.306000 269 torch/utils/cpp_extension.py:2425] If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'] to specific architectures.
[W1022 06:03:04.268878543 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[W1022 06:03:22.923950021 socket.cpp:200] [c10d] The hostname of the client socket cannot be retrieved. err=-3
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
(EngineCore_DP0 pid=269) INFO 10-22 06:03:22 [parallel_state.py:1208] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
(EngineCore_DP0 pid=269) INFO 10-22 06:03:22 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore_DP0 pid=269) INFO 10-22 06:03:25 [gpu_model_runner.py:2602] Starting to load model /data/models/Qwen3-VL-2B-Instruct...
(EngineCore_DP0 pid=269) INFO 10-22 06:03:26 [gpu_model_runner.py:2634] Loading model from scratch...
(EngineCore_DP0 pid=269) INFO 10-22 06:03:26 [cuda.py:366] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.06it/s]
(EngineCore_DP0 pid=269) INFO 10-22 06:03:28 [default_loader.py:267] Loading weights took 1.15 seconds
(EngineCore_DP0 pid=269) INFO 10-22 06:03:28 [gpu_model_runner.py:2653] Model loading took 4.2374 GiB and 2.352086 seconds
(EngineCore_DP0 pid=269) INFO 10-22 06:03:29 [gpu_model_runner.py:3344] Encoder cache will be initialized with a budget of 153600 tokens, and profiled with 1 video items of the maximum feature size.
(EngineCore_DP0 pid=269) INFO 10-22 06:03:46 [backends.py:548] Using cache directory: /root/.cache/vllm/torch_compile_cache/e2beb21e0e/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=269) INFO 10-22 06:03:46 [backends.py:559] Dynamo bytecode transform time: 6.84 s
(EngineCore_DP0 pid=269) INFO 10-22 06:03:53 [backends.py:197] Cache the graph for dynamic shape for later use
(EngineCore_DP0 pid=269) INFO 10-22 06:04:21 [backends.py:218] Compiling a graph for dynamic shape takes 35.11 s
(EngineCore_DP0 pid=269) INFO 10-22 06:04:26 [monitor.py:34] torch.compile takes 41.96 s in total
(EngineCore_DP0 pid=269) INFO 10-22 06:04:27 [gpu_worker.py:298] Available KV cache memory: 67.70 GiB
(EngineCore_DP0 pid=269) INFO 10-22 06:04:28 [kv_cache_utils.py:1087] GPU KV cache size: 633,856 tokens
(EngineCore_DP0 pid=269) INFO 10-22 06:04:28 [kv_cache_utils.py:1091] Maximum concurrency for 32,768 tokens per request: 19.34x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|██████████| 67/67 [00:07<00:00,  8.58it/s]
Capturing CUDA graphs (decode, FULL): 100%|██████████| 35/35 [00:02<00:00, 12.77it/s]
(EngineCore_DP0 pid=269) INFO 10-22 06:04:39 [gpu_model_runner.py:3480] Graph capturing finished in 11 secs, took 0.67 GiB
(EngineCore_DP0 pid=269) INFO 10-22 06:04:39 [core.py:210] init engine (profile, create kv cache, warmup model) took 70.82 seconds
(APIServer pid=1) INFO 10-22 06:04:41 [api_server.py:1634] Supported_tasks: ['generate']
(APIServer pid=1) WARNING 10-22 06:04:41 [model.py:1389] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=1) INFO 10-22 06:04:41 [serving_responses.py:137] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=1) INFO 10-22 06:04:41 [serving_chat.py:139] Using default chat sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=1) INFO 10-22 06:04:41 [serving_completion.py:76] Using default completion sampling params from model: {'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}
(APIServer pid=1) INFO 10-22 06:04:41 [api_server.py:1912] Starting vLLM API server 0 on http://0.0.0.0:30001
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:34] Available routes are:
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /docs, Methods: HEAD, GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /redoc, Methods: HEAD, GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /health, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /load, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /ping, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /ping, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /tokenize, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /detokenize, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/models, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /version, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/chat/completions, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/completions, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/embeddings, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /pooling, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /classify, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /score, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/score, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/audio/translations, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /rerank, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v1/rerank, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /v2/rerank, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /invocations, Methods: POST
(APIServer pid=1) INFO 10-22 06:04:41 [launcher.py:42] Route: /metrics, Methods: GET
(APIServer pid=1) INFO:     Started server process [1]
(APIServer pid=1) INFO:     Waiting for application startup.
(APIServer pid=1) INFO:     Application startup complete.

3、调用方式

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
curl --location 'http://10.39.214.112:30001/v1/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer <YOUR_API_KEY>' \
--data '{
    "model": "Qwen3-VL-2B-Instruct",
    "stream": false,
    "max_tokens": 120,
    "temperature": 0.7,
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "hi"
        }
    ]
}'

img_01

4、调用日志

1
2
(APIServer pid=1) INFO 10-22 06:05:12 [chat_utils.py:560] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=1) INFO:     10.33.27.93:50362 - "POST /v1/chat/completions HTTP/1.1" 200 OK

5、返回参数

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
{
    "id": "chatcmpl-aeddb44591eb4205b76174b2b0316023",
    "object": "chat.completion",
    "created": 1761113112,
    "model": "Qwen3-VL-2B-Instruct",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "Hello! How can I help you today?",
                "refusal": null,
                "annotations": null,
                "audio": null,
                "function_call": null,
                "tool_calls": [],
                "reasoning_content": null
            },
            "logprobs": null,
            "finish_reason": "stop",
            "stop_reason": null,
            "token_ids": null
        }
    ],
    "service_tier": null,
    "system_fingerprint": null,
    "usage": {
        "prompt_tokens": 20,
        "total_tokens": 30,
        "completion_tokens": 10,
        "prompt_tokens_details": null
    },
    "prompt_logprobs": null,
    "prompt_token_ids": null,
    "kv_transfer_params": null
}
本文结束 感谢您的阅读