[Bug] An error occurred when running the deepseek r1 awq model

[luhairong11]

Checklist

• 1. I have searched related issues but cannot get the expected help.
• 2. The bug has not been fixed in the latest version.
• 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
• 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/sgl-project/sglang/discussions/new/choose Otherwise, it will be closed.
• 5. Please use English, otherwise it will be closed.

Describe the bug

Reproduction

python3 -m sglang.launch_server --model-path /data/chdmx/models/LLM_models/deepseek_ai/DeepSeek-R1-awq --tp 8 --trust-remote-code --dtype half

Environment

root@bms-schyjdmx03:/opt/package/sglang# python3 -m sglang.check_env
2025-02-26 02:42:04,341 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
/usr/local/lib/python3.10/dist-packages/pydantic/_internal/_config.py:345: UserWarning: Valid config keys have changed in V2:

• 'fields' has been removed
  warnings.warn(message, UserWarning)
  Python: 3.10.16 (main, Dec 4 2024, 08:53:37) [GCC 9.4.0]
  CUDA available: True
  GPU 0,1,2,3,4,5,6,7: NVIDIA A100 80GB PCIe
  GPU 0,1,2,3,4,5,6,7 Compute Capability: 8.0
  CUDA_HOME: /usr/local/cuda
  NVCC: Cuda compilation tools, release 12.4, V12.4.131
  CUDA Driver Version: 550.54.15
  PyTorch: 2.5.1+cu124
  sglang: 0.4.3.post2
  sgl_kernel: 0.0.3.post6
  flashinfer: 0.2.2
  triton: 3.1.0
  transformers: 4.48.3
  torchao: 0.8.0
  numpy: 1.26.4
  aiohttp: 3.11.11
  fastapi: 0.115.6
  hf_transfer: 0.1.9
  huggingface_hub: 0.27.1
  interegular: 0.3.3
  modelscope: 1.22.3
  orjson: 3.10.15
  packaging: 24.2
  psutil: 6.1.1
  pydantic: 2.10.5
  multipart: 0.0.20
  zmq: 26.2.0
  uvicorn: 0.34.0
  uvloop: 0.21.0
  vllm: 0.6.4.post1
  openai: 1.59.8
  anthropic: 0.43.1
  decord: 0.6.0
  NVIDIA Topology:
  GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 CPU Affinity NUMA Affinity GPU NUMA ID
  GPU0 X PIX PIX PIX SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A
  GPU1 PIX X PIX PIX SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A
  GPU2 PIX PIX X PIX SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A
  GPU3 PIX PIX PIX X SYS SYS SYS SYS SYS SYS 0-27,56-83 0 N/A
  GPU4 SYS SYS SYS SYS X PIX PIX PIX SYS SYS 28-55,84-111 1 N/A
  GPU5 SYS SYS SYS SYS PIX X PIX PIX SYS SYS 28-55,84-111 1 N/A
  GPU6 SYS SYS SYS SYS PIX PIX X PIX SYS SYS 28-55,84-111 1 N/A
  GPU7 SYS SYS SYS SYS PIX PIX PIX X SYS SYS 28-55,84-111 1 N/A
  NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS
  NIC1 SYS SYS SYS SYS SYS SYS SYS SYS SYS X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1

ulimit soft: 1048576

[luhairong11]

After upgrading the vllm version, the model is able to run, but the inference speed is extremely slow.