SGLang、vLLM与LMDeploy推理框架对比：架构特点、部署实践与性能差异分析

发表于 | 分类于 | 2.5k | 14 分钟

SGLang

SGLang 是一种基于图计算的推理加速框架，通过图结构优化并行计算，支持复杂任务处理和分布式部署，特别适合高并发、低延迟的企业级应用。

sglang==0.4.3.post2

基于sglang_20250303:py311（sglang==0.4.3.post2）首次部署

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
# 创建容器（sglang_20250303:py311中sglang==0.4.3.post2）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management_sglang --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models sglang_20250303:py311 /bin/bash

# view sglang version
pip list | grep sglang

# deploy Qwen2.5-72B-Instruct
nohup python -m sglang.launch_server \
       --model-path /models/Qwen2.5-72B-Instruct \
       --served-model-name Qwen2.5-72B-Instruct \
       --context-length 16384 \
       --mem-fraction-static 0.8 \
       --trust-remote-code \
       --host 0.0.0.0 \
       --port 8996 \
       --tp 8  \
       --disable-radix-cache \
       --api-key 669b12de160848509c3a0ba5d7704729 >> /mnt/qwen25.log 2>&1 &

sglang==0.4.2.post4

基于sglang_20250212:py311（**sglang==0.4.2.post4**）首次部署
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
# 创建容器（sglang_20250212:py311中sglang==0.4.2.post4）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management_sglang --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models sglang_20250212:py311 /bin/bash

# view sglang version
pip list | grep sglang

# deploy Qwen2.5-72B-Instruct
nohup python -m sglang.launch_server \
       --model-path /models/Qwen2.5-72B-Instruct \
       --served-model-name Qwen2.5-72B-Instruct \
       --context-length 16384 \
       --mem-fraction-static 0.8 \
       --trust-remote-code \
       --host 0.0.0.0 \
       --port 8996 \
       --grammar-backend xgrammar \
       --tp 8  \
       --disable-radix-cache \
       --api-key 669b12de160848509c3a0ba5d7704729 >> /mnt/qwen25.log 2>&1 &
### **sglang**参数调优 **(1).MLA** 版本：**0.4.3** FlashInfer MLA Attention 的开创性集成**为长上下文场景带来了 4 倍的性能提升，仅支持DeepSeek系列，此版本应使用**`--enable-flashinfer-mla --disable-radix-cache`
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 创建容器（sglang_20250303:py311中sglang==0.4.3.post2）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management_sglang --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models sglang_20250303:py311 /bin/bash

# deploy Qwen2.5-72B-Instruct
nohup python -m sglang.launch_server \
       --model-path /models/Qwen2.5-72B-Instruct \
       --served-model-name Qwen2.5-72B-Instruct \
       --context-length 16384 \
       --mem-fraction-static 0.8 \
       --trust-remote-code \
       --host 0.0.0.0 \
       --port 8996 \
       --grammar-backend xgrammar \
       --tp 8  \
       --enable-flashinfer-mla \
       --disable-radix-cache \
       --api-key 669b12de160848509c3a0ba5d7704729 >> /mnt/qwen25.log 2>&1 &

(2).默认启用flash attention 3进行分块预填充
版本：0.4.3
(3).MTP
The accept rate of the MTP module is very high (~1.9 avg accept length for draft 2 tokens, e.g. --speculative-num-steps 2 --speculative-eagle-topk 1 --speculative-num-draft-tokens 2).
We try to use it to draft more tokens and achieved better speedup. (2.5~3 avg accept length for draft 4 tokens for 2 steps, e.g. --speculative-num-steps 2 --speculative-eagle-topk 4 --speculative-num-draft-tokens 4)
版本：v0.4.3.post1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# 创建容器（sglang_20250303:py311中sglang==0.4.3.post2）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management_sglang --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models sglang_20250303:py311 /bin/bash


# deploy Qwen2.5-72B-Instruct
nohup python -m sglang.launch_server \
       --model-path /models/Qwen2.5-72B-Instruct \
       --served-model-name Qwen2.5-72B-Instruct \
       --context-length 16384 \
       --mem-fraction-static 0.8 \
       --trust-remote-code \
       --host 0.0.0.0 \
       --port 8996 \
       --grammar-backend xgrammar \
       --tp 8  \
       --speculative-algo NEXTN \
       --speculative-draft /sgl-workspace/DeepSeek-V3-nextn \
       --speculative-num-steps 2 \
       --speculative-eagle-topk 4 \
       --speculative-num-draft-tokens 4 \
       --disable-radix-cache \
       --api-key 669b12de160848509c3a0ba5d7704729 >> /mnt/qwen25.log 2>&1 &

(4).增加了对 FP8 的 torch.compile 支持，在线推理速度达到 50 token/s
版本：0.4.3
此版本应使用：--enable-torch-compile --torch-compile-max-bs 1

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
# 创建容器（sglang_20250303:py311中sglang==0.4.3.post2）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management_sglang --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models sglang_20250303:py311 /bin/bash

# deploy Qwen2.5-72B-Instruct
nohup python -m sglang.launch_server \
       --model-path /models/Qwen2.5-72B-Instruct \
       --served-model-name Qwen2.5-72B-Instruct \
       --context-length 16384 \
       --mem-fraction-static 0.8 \
       --trust-remote-code \
       --host 0.0.0.0 \
       --port 8996 \
       --tp 8  \
       --enable-torch-compile \
       --torch-compile-max-bs 1 \
       --disable-radix-cache \
       --api-key 669b12de160848509c3a0ba5d7704729 >> /mnt/qwen25.log 2>&1 &

vllm

vLLM 是一种高性能的 GPU 推理框架，通过 PagedAttention 技术优化 GPU 内存效率和吞吐量，适用于大规模高并发场景。

vllm==0.7.3

基于llm_management_20250224:py311（vllm==0.7.3）首次部署

1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建容器（llm_management_20250224:py311中vllm==0.7.3）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models llm_management_20250224:py311 /bin/bash

# 8卡启动Qwen2.5-72B-Instruct
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup python -m vllm.entrypoints.openai.api_server \
		--served-model-name Qwen2.5-72B-Instruct \
		--port 8996 \
		--model /models/Qwen2.5-72B-Instruct \
		--tensor-parallel-size 8 \
		--max-model-len 16384 \
		--gpu-memory-utilization 0.80 \
		--disable-log-stats \
		--api-key 669b12de160848509c3a0ba5d7704729 > Qwen2.5-72B-Instruct.log 2>&1 &

vllm==0.6.0

基于llm_management_20240910:py311（vllm==0.6.0）首次部署

1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建容器（llm_management_20240910:py311中vllm==0.6.0）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models llm_management_20240910:py311 /bin/bash

# 8卡启动Qwen2.5-72B-Instruct
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup python -m vllm.entrypoints.openai.api_server \
    --served-model-name Qwen2.5-72B-Instruct \
    --port 8996 \
    --model /models/Qwen2.5-72B-Instruct \
    --tensor-parallel-size 8 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.80 \
    --disable-log-stats \
    --api-key 669b12de160848509c3a0ba5d7704729 > Qwen2.5-72B-Instruct.log 2>&1 &

vllm参数调优

基于llm_management_20250224:py311（vllm==0.7.3）首次部署
To use vLLM V1:

Install the latest version of vLLM with pip install vllm –upgrade.
Set the environment variable export VLLM_USE_V1=1.
Use vLLM’s Python API or OpenAI-compatible server (vllm serve <model-name>). You don’t need any change to the existing API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# 创建容器（llm_management_20250224:py311中vllm==0.7.3）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models llm_management_20250224:py311 /bin/bash

# 8卡启动Qwen2.5-72B-Instruct
export VLLM_USE_V1=1
export VLLM_WORKER_MULTIPROC_METHOD=spawn
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7 nohup python -m vllm.entrypoints.openai.api_server \
		--served-model-name Qwen2.5-72B-Instruct \
		--port 8996 \
		--model /models/Qwen2.5-72B-Instruct \
		--tensor-parallel-size 8 \
		--max-model-len 16384 \
		--gpu-memory-utilization 0.80 \
		--disable-log-stats \
		--api-key 669b12de160848509c3a0ba5d7704729 > Qwen2.5-72B-Instruct.log 2>&1 &

LMDeploy

LMDeploy 是一个高效的大模型推理框架，提供低延迟和高吞吐量的推理能力，支持模型量化和多模态任务部署，适用于企业级实时应用。

lmdeploy==0.7.1

1
2
3
4
5
6
7
8
9
10
11
12
# 创建容器（lmdeploy_20250304:py311中lmdeploy==0.7.1）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models lmdeploy_20250304:py311 /bin/bash

# 8卡启动Qwen2.5-72B-Instruct
lmdeploy serve api_server /models/Qwen2.5-72B-Instruct \
    --model-name Qwen2.5-72B-Instruct \
    --server-name 0.0.0.0 \
    --server-port 8996 \
    --tp 8 \
    --session-len 16384 \
    --cache-max-entry-count 0.8 \
    --api-key 669b12de160848509c3a0ba5d7704729

lmdelploy参数调优


(1).max-batch-size
此版本应使用--max-batch-size 32

1
2
3
4
5
6
7
8
9
10
11
12
13
# 创建容器（lmdeploy_20250304:py311中lmdeploy==0.7.1）
sudo docker run --gpus all -it -d --network llm-management-network --name llm_management --ipc=host -P -p 8900-8997:8900-8997 -v /data/models:/models lmdeploy_20250304:py311 /bin/bash

# 8卡启动Qwen2.5-72B-Instruct
lmdeploy serve api_server /models/Qwen2.5-72B-Instruct \
    --model-name Qwen2.5-72B-Instruct \
    --server-name 0.0.0.0 \
    --server-port 8996 \
    --tp 8 \
    --session-len 16384 \
    --cache-max-entry-count 0.8 \
    --api-key 669b12de160848509c3a0ba5d7704729 \
    --max-batch-size 32

测试结果

以sglang、vllm方式启用，openai client调用，测试qwen模型并发效果

test_qwen_concurrent.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
# -*- coding: utf-8 -*-
"""
@Time    : 2025/2/26 10:45
@Author  : gkm0120
@File    : test_qwen_concurrent.py
@Software: PyCharm
"""
# 以sglang、vllm方式启用，openai client调用，测试qwen模型并发效果
import concurrent.futures
from datetime import datetime

from loguru import logger
from openai import OpenAI

start_time = datetime.now()
response_start_time_list = []
response_end_time_list = []

# Initialize OpenAI client
client = OpenAI(
    api_key="669b12de160848509c3a0ba5d7704729",
    base_url="http://10.39.214.107:8996/v1",
)

def send_post_request(test_prompt, model, stream=False, max_tokens=512):
    """
    Send POST request
    :param test_prompt: The test prompt
    :param model: Model name
    :param stream: Whether to stream the output
    :param max_tokens: Maximum tokens in the output
    :return:
    """
    response_start_time = datetime.now()
    logger.info("response starting")
    collected_messages = []
    try:
        response = client.chat.completions.create(
            model=model,
            messages=[{"role": "user", "content": test_prompt}],
            temperature=0.7,
            stream=True,  # Default false, can be true
            max_tokens=max_tokens
        )

        if not stream:
            logger.info(response)
        if stream:
            for index, chunk in enumerate(response):
                if not chunk.choices:
                    continue
                chunk_message = chunk.choices[0].delta
                if not chunk_message:
                    continue
                    
                 if index == 1:
                    logger.info(chunk_message.content)
                    first_token_return_time = datetime.now()
                    logger.info(f"first token return time: {first_token_return_time - response_start_time}")
                    response_start_time_list.append((first_token_return_time - response_start_time).total_seconds())

                if chunk_message.content is not None:
                    collected_messages += chunk_message.content

            response_end_time = datetime.now()
            logger.info(f"response running time: {response_end_time - response_start_time}")
            logger.info(f"collected messages: {''.join(collected_messages)}")
            response_end_time_list.append((response_end_time - response_start_time).total_seconds())
    except Exception as e:
        logger.info(f"Exception: {e}")
        
def run_concurrent_requests(num_requests, model, stream, max_tokens, prompts):
    """
    Execute concurrent tasks
    :param num_requests: Number of concurrent requests
    :param model: Model name
    :param stream: Whether to stream output
    :param max_tokens: Maximum tokens in the output
    :param prompts: Prompt list for concurrent requests
    :return:
    """
    with concurrent.futures.ThreadPoolExecutor(max_workers=num_requests) as executor:
        # Start concurrent tasks
        futures = [executor.submit(send_post_request, prompts[i], model, stream, max_tokens) for i in range(num_requests)]
        # Wait for all tasks to complete
        concurrent.futures.wait(futures)

    end_time = datetime.now()
    logger.info(f"end_time - start_time: {end_time - start_time}\n")

if __name__ == "__main__":
    from transformers import AutoTokenizer
    model_dir = "./models/Qwen2.5-72B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)

    # Test parameters: number of concurrent requests, model, stream, max_tokens
    test_num_concurrent_requests = 5
    test_input_prompt_token = 512
    test_model = "Qwen2.5-72B-Instruct"
    test_stream = True
    test_max_tokens = 128

    # Generate 5 test prompts, each of length 512, with tokenizer
    from utils.generate_prompt_util import sample_sonnet_requests
    input_requests = sample_sonnet_requests(
        dataset_path="./datasets/sonnet.txt",
        num_requests=test_num_concurrent_requests,
        input_len=test_input_prompt_token,
        output_len=test_max_tokens,
        prefix_len=50,
        tokenizer=tokenizer,
    )
    test_prompts = [prompt_formatted for prompt, prompt_formatted, prompt_len, output_len in input_requests]

    # Run concurrent requests
    run_concurrent_requests(test_num_concurrent_requests, test_model, test_stream, test_max_tokens, test_prompts)
    response_start_time_list = sorted(response_start_time_list)
    response_end_time_list = sorted(response_end_time_list)

    # Test results
    logger.info(f"response_time_to_first_token_list: {response_start_time_list}")
    response_start_time_average = sum(response_start_time_list) / len(response_start_time_list)
    logger.info(f"response_time_to_first_token_average: {response_start_time_average}")
    logger.info(f"response_end_time_list: {response_end_time_list}")
    response_end_time_average = sum(response_end_time_list) / len(response_end_time_average)
    logger.info(f"response_end_time_average: {response_end_time_average}\n")
    average_completion_time = response_end_time_average - response_start_time_average
    logger.info(f"average_completion_time: {average_completion_time}")
    logger.info(f"average_generate_speed: {test_max_tokens / average_completion_time}")
generate_prompt_util.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
# -*- coding: utf-8 -*-
"""
@Time    : 2025/2/24 11:51
@Author  : gkm0120
@File    : generate_prompt_util.py
@Software: PyCharm
"""
import random
from typing import Tuple, List

from loguru import logger
from transformers import PreTrainedTokenizerBase


def sample_sonnet_requests(
        dataset_path: str,
        num_requests: int,
        input_len: int,
        output_len: int,
        prefix_len: int,
        tokenizer: PreTrainedTokenizerBase,
) -> list[tuple[str, int, int]]:
    assert (
            input_len > prefix_len
    ), "'args.sonnet-input-len' must be greater than 'args.prefix-input-len'."

    # Load the dataset.
    with open(dataset_path) as f:
        poem_lines = f.readlines()

    # Tokenize the poem lines.
    poem_token_ids = tokenizer(poem_lines).input_ids
    average_poem_len = sum(
        len(token_ids) for token_ids in poem_token_ids) / len(poem_token_ids)

    # Base prefix for all requests.
    base_prompt = "Pick as many lines as you can from these poem lines:\n"
    base_message = [{
        "role": "user",
        "content": base_prompt,
    }]
    base_prompt_formatted = tokenizer.apply_chat_template(
        base_message, add_generation_prompt=True, tokenize=False)
    base_prompt_offset = len(tokenizer(base_prompt_formatted).input_ids)

    assert (
            input_len > base_prompt_offset
    ), f"Please set 'args.sonnet-input-len' higher than {base_prompt_offset}."
    num_input_lines = round(
        (input_len - base_prompt_offset) / average_poem_len)

    # First approximately `prefix_len` number of tokens in the
    # prompt are fixed poem lines.
    assert (
            prefix_len > base_prompt_offset
    ), f"Please set 'args.sonnet-prefix-len' higher than {base_prompt_offset}."

    num_prefix_lines = round(
        (prefix_len - base_prompt_offset) / average_poem_len)
    prefix_lines = poem_lines[:num_prefix_lines]

    # Sample the rest of lines per request.
    sampled_requests: List[Tuple[str, int, int]] = []
    for _ in range(num_requests):
        sampled_lines = "".join(
            prefix_lines +
            random.sample(poem_lines, num_input_lines - num_prefix_lines))

        prompt = f"{base_prompt}{sampled_lines}"
        message = [
            {
                "role": "user",
                "content": prompt,
            },
        ]
        prompt_formatted = tokenizer.apply_chat_template(
            message, add_generation_prompt=True, tokenize=False)
        prompt_len = len(tokenizer(prompt_formatted).input_ids)
        sampled_requests.append((prompt, prompt_formatted, prompt_len, output_len))

    return sampled_requests


if __name__ == "__main__":
    # Generate 20 random test prompts, each of length 512
    from transformers import AutoTokenizer
    model_dir = "./tests/models/Qwen2.5-72B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)

    input_requests = sample_sonnet_requests(
        dataset_path="./tests/datasets/sonnet.txt",
        num_requests=5,
        input_len=512,
        output_len=128,
        prefix_len=50,
        tokenizer=tokenizer,
    )
    test_prompts = [prompt_formatted for prompt, prompt_formatted, prompt_len, output_len in input_requests]
    logger.info(test_prompts)
### 框架对比 **不同框架对比结果** ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_a917f113.png) **同一框架不同版本对比结果** ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_5bdbd442.png) ### **SGLang测试结果** ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_eaa3606a.png) ### **vLLM测试结果** ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_0dda23d9.png) ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_5cb83b74.png) ### **LMDeploy测试结果** ![](https://cdn.jsdelivr.net/gh/gkm0120/CDN/img/notion_90741d4a.png)
本文结束 感谢您的阅读