使用Transformers推理 使用Chinese-Alpaca-33B模型时报错.

[SimTsai]

使用的是已经合并过的HF 33B模型.

python scripts/inference/inference_hf.py \
--base_model ~/chinese_alpaca_33b_bin \
--with_prompt --interactive

The model weights are not tied. Please use the tie_weights method before using the infer_auto_device function.
Traceback (most recent call last):
File "/home/github/ymcui/Chinese-LLaMA-Alpaca/scripts/inference/inference_hf.py", line 62, in
base_model = LlamaForCausalLM.from_pretrained(
File "/home/anaconda3/envs/Chinese-LLaMA-Alpaca/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2795, in from_pretrained
) = cls._load_pretrained_model(
File "/home/anaconda3/envs/Chinese-LLaMA-Alpaca/lib/python3.10/site-packages/transformers/modeling_utils.py", line 2889, in _load_pretrained_model
raise ValueError(
ValueError: The current device_map had weights offloaded to the disk. Please provide an offload_folder for them. Alternatively, make sure you have safetensors installed if the model you are using offers the weights in this format.

[airaria]

机器上有GPU吗？GPU显存多少？

[SimTsai]

有GPU, 4G的...是因为显存太低了么?

[airaria]

显存太低了。建议4bit量化后用llama.cpp加载

[SimTsai]

4bit量化后的llama.cpp我试过了, 是可以跑起来,但是好像也不走GPU,感觉都是CPU在做推理.并且结果也不太理想.
所以想换成Transformers试一试, 如果需要跑 33B 的话推荐需要多大显存呢?

[ymcui]

1. 不走GPU：你要看是否是和cuBLAS一起编译过的，-ngl指定offload多少层到GPU上。
2. 效果不理想：这个太主观，首先你要排查合并后模型的SHA256是否一致。另外，建议你看看我们的example，给定相同prompt的情况下结果不应该有太大出入，多试几个不同类型的prompt排查是不是模型本身有问题。

[SimTsai]

1. 是cuBLAS编译版本的lama.cpp, 运行时加过 -ngl 5, 但是推理的时候基本上CPU是满的, GPU是躺平状态.

./main -m zh-models/33B/ggml-model-q4_0.bin --color -f prompts/alpaca-cn.txt -ins -c 2048 --temp 0.2 -n 256 --repeat_penalty 1.1 -r "User:" -ngl 5
main: build = 711 (16b9cd1)
main: seed  = 1687677163
ggml_init_cublas: found 1 CUDA devices:
  Device 0: NVIDIA GeForce GTX 1650
llama.cpp: loading model from zh-models/33B/ggml-model-q4_0.bin
llama_model_load_internal: format     = ggjt v3 (latest)
llama_model_load_internal: n_vocab    = 49954
llama_model_load_internal: n_ctx      = 2048
llama_model_load_internal: n_embd     = 6656
llama_model_load_internal: n_mult     = 256
llama_model_load_internal: n_head     = 52
llama_model_load_internal: n_layer    = 60
llama_model_load_internal: n_rot      = 128
llama_model_load_internal: ftype      = 2 (mostly Q4_0)
llama_model_load_internal: n_ff       = 17920
llama_model_load_internal: n_parts    = 1
llama_model_load_internal: model size = 30B
llama_model_load_internal: ggml ctx size =    0.13 MB
llama_model_load_internal: using CUDA for GPU acceleration
llama_model_load_internal: mem required  = 18531.29 MB (+ 3124.00 MB per state)
llama_model_load_internal: allocating batch_size x 1 MB = 512 MB VRAM for the scratch buffer
llama_model_load_internal: offloading 5 repeating layers to GPU
llama_model_load_internal: offloaded 5/63 layers to GPU
llama_model_load_internal: total VRAM used: 1948 MB