GPTQ 量化详解：基于近似二阶信息的一次性权重量化

讲解 GPTQ 这一面向大语言模型的一次性权重量化方法：利用近似二阶信息逐层压缩权重，并结合 Transformers 加载 GPTQ-Int8 模型、用 AutoGPTQ 量化自定义模型以及配合 vLLM 部署的完整示例。

| **[GitHub](https://github.com/AutoGPTQ/AutoGPTQ) | [Paper ](https://arxiv.org/abs/2210.17323)|**

简介

GPTQ是一种类似GPT的LLM的量化方法，它使用基于近似二阶信息的一次性权重量化。

https://zhuanlan.zhihu.com/p/2376060096

Transformer

from transformers import AutoModelForCausalLM, AutoTokenizer
device = "cuda" # the device to load the model onto

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen1.5-7B-Chat-GPTQ-Int8", # the quantized model
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-7B-Chat-GPTQ-Int8")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt").to(device)

generated_ids = model.generate(
    model_inputs.input_ids,
    max_new_tokens=512
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]

vLLM

vLLM 支持 GPTQ，这意味着您可以直接使用我们提供的 GPTQ 模型或通过 vLLM 进行 AutoGPTQ 训练的模型。

python -m vllm.entrypoints.openai.api_server --model Qwen/Qwen1.5-7B-Chat-GPTQ-Int8

curl http://localhost:8000/v1/chat/completions  -H "Content-Type: application/json" -d '{
   "model": "Qwen/Qwen1.5-7B-Chat-GPTQ-Int8",
   "messages": [
   {"role": "system", "content": "You are a helpful assistant."},
   {"role": "user", "content": "Tell me something about large language models."}
   ],
   }'

或者使用 python client 调用：

from openai import OpenAI
# Set OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

chat_response = client.chat.completions.create(
    model="Qwen/Qwen1.5-7B-Chat-GPTQ-Int8",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Tell me something about large language models."},
    ]
)
print("Chat response:", chat_response)

AutoGPTQ 量化自定义模型

建议通过源代码安装来安装最新版本的软件包：

git clone https://github.com/AutoGPTQ/AutoGPTQ
cd AutoGPTQ
pip install -e .

构建自己的AWQ量化模型，并使用训练数据进行校准：

from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
from transformers import AutoTokenizer

# Specify paths and hyperparameters for quantization
model_path = "your_model_path"
quant_path = "your_quantized_model_path"
quantize_config = BaseQuantizeConfig(
    bits=8, # 4 or 8
    group_size=128,
    damp_percent=0.01,
    desc_act=False,  # set to False can significantly speed up inference but the perplexity may slightly bad
    static_groups=False,
    sym=True,
    true_sequential=True,
    model_name_or_path=None,
    model_file_base_name="model"
)
max_len = 8192

# Load your tokenizer and model with AutoGPTQ
# To learn about loading model to multiple GPUs,
# visit https://github.com/AutoGPTQ/AutoGPTQ/blob/main/docs/tutorial/02-Advanced-Model-Loading-and-Best-Practice.md
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoGPTQForCausalLM.from_pretrained(model_path, quantize_config)

如果想在多个 GPU 上加载模型，则需要使用 max_memory 而不是 device_map。比如：

model = AutoGPTQForCausalLM.from_pretrained(
    model_path,
    quantize_config,
    max_memory={i:"20GB" for i in range(4)}
)

需要准备数据以进行校准。将样本放入一个列表中，其中每个样本都是一段文本。由于直接使用微调数据来进行校准，所以需要使用ChatML模板对其进行格式化：

import torch

data = []
for msg in messages:
    text = tokenizer.apply_chat_template(msg, tokenize=False, add_generation_prompt=False)
    model_inputs = tokenizer([text])
    input_ids = torch.tensor(model_inputs.input_ids[:max_len], dtype=torch.int)
    data.append(dict(input_ids=input_ids, attention_mask=input_ids.ne(tokenizer.pad_token_id)))

其中每个 msg 是一个典型的聊天消息，如下所示：

[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Tell me who you are."},
    {"role": "assistant", "content": "I am a large language model named Qwen..."}
]

最后保存自己的模型：

model.save_quantized(quant_path, use_safetensors=True)
tokenizer.save_pretrained(quant_path)