报错

 File "/opt/conda/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/awq.py", line 6, in <module>
    from vllm._C import ops
ImportError: libcudart.so.12: cannot open shared object file: No such file or directory

参考：
https://hub.yzuu.cf/vllm-project/vllm/issues/1716
https://hub.yzuu.cf/vllm-project/vllm/issues/1718

解决方案

Hi @phatjkk, please install vLLM + CUDA 11.8 following the instructions here.
vLLM v0.2.2 depends on PyTorch 2.1 and CUDA 12.1. This might cause error if you environment does not have CUDA 12.1.

FROM pytorch/pytorch:2.1.2-cuda12.1-cudnn8-devel