vLLM官方Benchmark工具使用指南：延迟、前缀缓存、服务端与吞吐量四类压测脚本实战

发表于 | 分类于 | 667 | 3 分钟

vllm/benchmarks at v0.6.0 · vllm-project/vllm

defog/llama-3-sqlcoder-8b · Hugging Face

huggingface.co

1、benchmark_latency.py

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
1、启动服务（会占用显存）
python benchmark_latency.py \
    --model /models/llama-3-sqlcoder-8b/ \
    --tensor-parallel-size 1 
    --gpu-memory-utilization 0.9

# 2、执行后的结果
Avg latency: 1.58441203776747 seconds
10% percentile latency: 1.578821041993797 seconds
25% percentile latency: 1.5793092143721879 seconds
50% percentile latency: 1.579846273176372 seconds
75% percentile latency: 1.5803136085160077 seconds
90% percentile latency: 1.5815249111503362 seconds
99% percentile latency: 1.6747117711603643 seconds

2、benchmark_prefix_caching.py

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 1、启动服务（会占用显存）
# This command samples 20 prompts with input lengths
# between 128 and 256 tokens from the ShareGPT dataset,
# then replicates each prompt 5 times.

python benchmark_prefix_caching.py \
    --model /models/llama-3-sqlcoder-8b/ \
    --dataset-path /models/ShareGPT_V3_unfiltered_cleaned_split.json \
    --enable-prefix-caching \
    --num-prompts 20 \
    --repeat-count 5 \
    --input-length-range 128:256

# 2、执行后的结果
Processed prompts: 100%|████████████████████| 100/100 [00:00<00:00, 109.16it/s, est. speed input: 21380.07 toks/s, output: 1009.77 toks/s]
cost time 0.9721899032592773
------start generating------
Processed prompts: 100%|████████████████████| 100/100 [00:00<00:00, 302.34it/s, est. speed input: 59221.85 toks/s, output: 2869.56 toks/s]
cost time 0.3786342144012451

3、benchmark_serving.py

text
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
# 1、启动vllm服务（以模型llama-3-sqlcoder-8b为例）
CUDA_VISIBLE_DEVICES=7 python -m vllm.entrypoints.openai.api_server  \
    --port 8000 
    --model /models/llama-3-sqlcoder-8b 
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --gpu-memory-utilization 0.28  \
    --disable-log-stats \
    --enable-prefix-caching

# 2、执行性能评估脚本 （随机从 ShareGPT 提供的用户和 GPT 对话数据当中，筛选 num-prompts 个问题进行测试）
python benchmark_serving.py \
    --model /models/llama-3-sqlcoder-8b \
    --dataset-path /models/ShareGPT_V3_unfiltered_cleaned_split.json \
    --request-rate 1 \
    --num-prompts 1000 \
    --seed 10

# 3、执行后的结果
    ============ Serving Benchmark Result ============
    Successful requests:                     1000      
    Benchmark duration (s):                  979.83    
    Total input tokens:                      223437    
    Total generated tokens:                  182497    
    Request throughput (req/s):              1.02      
    Output token throughput (tok/s):         186.25    
    Total Token throughput (tok/s):          414.29    
    ---------------Time to First Token----------------
    Mean TTFT (ms):                          55.83     
    Median TTFT (ms):                        51.36     
    P99 TTFT (ms):                           122.37    
    -----Time per Output Token (excl. 1st token)------
    Mean TPOT (ms):                          14.61     
    Median TPOT (ms):                        14.45     
    P99 TPOT (ms):                           19.02     
    ---------------Inter-token Latency----------------
    Mean ITL (ms):                           14.48     
    Median ITL (ms):                         14.04     
    P99 ITL (ms):                            29.98     
    ================================================== 
    
    
# 1、执行性能评估脚本 （随机从 ShareGPT 提供的用户和 GPT 对话数据当中，筛选 num-prompts 个问题进行测试）
python benchmark_serving.py \
    --model /models/llama-3-sqlcoder-8b \
    --dataset-path /models/ShareGPT_V3_unfiltered_cleaned_split.json \
    --request-rate 5 \
    --num-prompts 1000 \
    --seed 10

# 2、执行后的结果
============ Serving Benchmark Result ============
Successful requests:                     1000
Benchmark duration (s):                  207.66
Total input tokens:                      223437
Total generated tokens:                  182854
Request throughput (req/s):              4.82
Output token throughput (tok/s):         880.53
Total Token throughput (tok/s):          1956.50
---------------Time to First Token----------------
Mean TTFT (ms):                          70.06     
Median TTFT (ms):                        64.34     
P99 TTFT (ms):                           173.60    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          24.84     
Median TPOT (ms):                        24.65     
P99 TPOT (ms):                           43.54     
---------------Inter-token Latency----------------
Mean ITL (ms):                           24.15     
Median ITL (ms):                         21.01     
P99 ITL (ms):                            80.96     
==================================================

4、benchmark_throughput.py

text
1
2
3
4
5
6
7
8
9
10
# 1、启动服务（会占用显存）
python benchmark_throughput.py \
    --dataset /models/ShareGPT_V3_unfiltered_cleaned_split.json \
    --model /models/llama-3-sqlcoder-8b/ \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.9

# 2、执行后的结果
Processed prompts: 100%|████████████████████| 1000/1000 [00:59<00:00, 16.73it/s, est. speed input: 3600.29 toks/s, output: 3318.26 toks/s]
Throughput: 16.57 requests/s, 6851.75 tokens/s
本文结束 感谢您的阅读