2080Ti安装vLLM完整教程：CUDA/PyTorch/Anaconda环境搭建、Qwen1.5-14B部署与常见报错排查

CUDA安装教程

CUDA 安装

# 检查 NVIDIA 驱动是否正常工作
nvidia-smi

# 更新系统并安装依赖包
sudo apt update
sudo apt install gcc g++ make
sudo apt install libglu1-mesa libxi-dev libxmu-dev libglu1-mesa-dev freeglut3-dev

# 下载并安装 CUDA
wget https://developer.download.nvidia.com/compute/cuda/12.1.0/local_installers/cuda_12.1.0_530.30.02_linux.run
sudo sh cuda_12.1.0_530.30.02_linux.run

# 验证 CUDA 安装
ls -l /usr/local | grep cuda
nvcc -V

# 配置环境变量
export PATH=$PATH:/usr/local/cuda-12.1/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.1/lib64
source ~/.bashrc

# 安装 cuDNN 加速计算
tar -xvf cudnn-linux-x86_64-8.9.6.50_cuda12-archive.tar.xz
sudo cp cudnn-linux-x86_64-8.9.6.50_cuda12-archive/include/cudnn.h /usr/local/cuda-12.1/include
sudo cp cudnn-linux-x86_64-8.9.6.50_cuda12-archive/lib/libcudnn* /usr/local/cuda-12.1/lib64
sudo chmod a+r /usr/local/cuda-12.1/include/cudnn.h

PyTorch 安装

# 自动安装 PyTorch
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

# 手动安装 PyTorch
pip install torch-2.1.0+cu121-cp38-cp38-linux_x86_64.whl
pip uninstall torchvision
pip install torchvision-0.16.0+cu121-cp38-cp38-linux_x86_64.whl

# 验证安装
python3
import torch
print(torch.cuda.is_available())

Anaconda3安装

登录23服务器

docker exec id /bin/bash

sudo wget https://repo.continuum.io/archive/Anaconda3-2022.10-Linux-x86_64.sh
sudo sh Anaconda3-2022.10-Linux-x86_64.sh
sudo vim ~/.bashrc

将最后一行加上

export PATH=$PATH:/data/young/anaconda3/bin
source ~/.bashrc
conda -V

更新apt

sudo apt-get update
sudo apt-get install gcc

服务部署和调用

安装vllm

pip install vllm

安装git lfs

wget https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh
bash script.deb.sh
apt-get install git-lfs

下载模型

git lfs https://hf-mirror.com/Qwen/Qwen1.5-14B-Chat

wget https://hf-mirror.com/Qwen/Qwen1.5-14B-Chat/resolve/main/model-0000\{1..8\}-of-00008.safetensors?download=true

重命名模型

import os
 
directory = '/mnt/young/models/Qwen1.5-14B-Chat'
 
for filename in os.listdir(directory):
    if filename.endswith('download=true'):
        new_filename = filename.replace('.safetensors?download=true', '.safetensors')
        os.rename(os.path.join(directory, filename), os.path.join(directory, new_filename))

服务启动

bug1:

Total number of attention heads (40) must be divisible by tensor parallel size (7)

bug2:

ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your NVIDIA GeForce RTX 2080 Ti GPU has compute capability 7.5. You can use float16 instead by explicitly setting thedtype flag in CLI, for example: –dtype=half.

bug3:

ValueError: The model’s max seq len (32768) is larger than the maximum number of tokens that can be stored in KV cache (3200). Try increasing gpu_memory_utilization or decreasing max_model_len when initializing the engine.

text

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m vllm.entrypoints.openai.api_server --served-model-name Qwen1.5-14B-Chat --model /mnt/young/models/Qwen1.5-14B-Chat/ --tensor-parallel-size 4 --dtype float16 --max-model-len 4096

服务调用

postman调用

text

curl --location 'http://10.39.101.123:8000/v1/chat/completions' \
--header 'Content-Type: application/json' \
--data '{
    "model": "Qwen1.5-14B-Chat",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "你好，你是谁？"
        }
    ]
}'

python调用

import requests
import json
url = "http://10.39.101.123:8000/v1/chat/completions"

payload = json.dumps({
  "model": "Qwen1.5-14B-Chat",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "你好，你是谁？"
    }
  ]
})
headers = {
  'Content-Type': 'application/json'
}

response = requests.request("POST", url, headers=headers, data=payload)
print(response.text)