some questions while reproduction #14
Description
Hi @Yangsenqiao , Thanks for your execellent work!
I'm a first-year graduate student and I used the trained models to evaluate and the results have no difference with the one you provided in paper.
Then i tried to reproduce the training process.At first ,i used tensordict<0.6(0.5.0) and got some import errors,then i changed the version to 0.7.2.
it seems worked.but there is a waring:verl 0.2.0.dev0 has requirement tensordict<0.6, but you have tensordict 0.7.2. The same warning occued when i changed the version of transfomers to 4.51.0(vllm 0.8.5 requires transformers>=4.51.1, but you have transformers 4.50.0 which is incompatible.)
Question 1:Is this waring has any influence on the trraining ?
The detailed environments I use are as follows:
accelerate 1.10.1
aiohappyeyeballs 2.6.1
aiohttp 3.13.1
aiosignal 1.4.0
airportsdata 20250909
annotated-types 0.7.0
antlr4-python3-runtime 4.9.3
anyio 3.7.1
astor 0.8.1
attrs 25.4.0
blake3 1.0.8
cachetools 6.2.1
certifi 2025.10.5
charset-normalizer 3.4.4
click 8.2.1
cloudpickle 3.1.1
codetiming 1.4.0
compressed-tensors 0.9.3
cupy-cuda12x 13.6.0
datasets 4.0.0
Deprecated 1.2.18
depyf 0.18.0
dill 0.3.8
diskcache 5.6.3
distro 1.9.0
dnspython 2.8.0
einops 0.8.1
email-validator 2.3.0
fastapi 0.115.14
fastapi-cli 0.0.7
fastrlock 0.8.3
filelock 3.20.0
flash-attn 2.7.1.post1
frozenlist 1.8.0
fsspec 2025.3.0
gguf 0.17.1
gitdb 4.0.12
GitPython 3.1.45
googleapis-common-protos 1.70.0
grpcio 1.75.1
h11 0.12.0
hf-xet 1.1.10
httpcore 0.15.0
httptools 0.7.1
httpx 0.23.0
huggingface-hub 0.35.3
hydra-core 1.3.2
idna 3.11
importlib_metadata 8.0.0
interegular 0.3.3
Jinja2 3.1.6
jiter 0.11.1
jsonschema 4.25.1
jsonschema-specifications 2025.9.1
lark 1.2.2
llguidance 0.7.30
llvmlite 0.44.0
lm-format-enforcer 0.10.12
markdown-it-py 4.0.0
MarkupSafe 3.0.3
mathruler 0.1.0
mdurl 0.1.2
mistral_common 1.8.5
mpmath 1.3.0
msgpack 1.1.2
msgspec 0.19.0
multidict 6.7.0
multiprocess 0.70.16
nest-asyncio 1.6.0
networkx 3.5
ninja 1.13.0
numba 0.61.2
numpy 2.2.6
nvidia-cublas-cu12 12.4.5.8
nvidia-cuda-cupti-cu12 12.4.127
nvidia-cuda-nvrtc-cu12 12.4.127
nvidia-cuda-runtime-cu12 12.4.127
nvidia-cudnn-cu12 9.1.0.70
nvidia-cufft-cu12 11.2.1.3
nvidia-curand-cu12 10.3.5.147
nvidia-cusolver-cu12 11.6.1.9
nvidia-cusparse-cu12 12.3.1.170
nvidia-cusparselt-cu12 0.6.2
nvidia-nccl-cu12 2.21.5
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu12 12.4.127
omegaconf 2.3.0
openai 2.5.0
opencv-python-headless 4.12.0.88
opentelemetry-api 1.26.0
opentelemetry-exporter-otlp 1.26.0
opentelemetry-exporter-otlp-proto-common 1.26.0
opentelemetry-exporter-otlp-proto-grpc 1.26.0
opentelemetry-exporter-otlp-proto-http 1.26.0
opentelemetry-proto 1.26.0
opentelemetry-sdk 1.26.0
opentelemetry-semantic-conventions 0.47b0
opentelemetry-semantic-conventions-ai 0.4.13
orjson 3.11.3
outlines 0.1.11
outlines_core 0.1.26
packaging 25.0
pandas 2.3.3
partial-json-parser 0.2.1.1.post6
peft 0.17.1
pillow 12.0.0
pip 25.2
platformdirs 4.5.0
prometheus_client 0.23.1
prometheus-fastapi-instrumentator 7.1.0
propcache 0.4.1
protobuf 4.25.8
psutil 7.1.1
py-cpuinfo 9.0.0
pyarrow 16.0.0
pybind11 3.0.1
pycountry 24.6.1
pydantic 2.12.3
pydantic_core 2.41.4
pydantic-extra-types 2.10.6
Pygments 2.19.2
pylatexenc 2.10
python-dateutil 2.9.0.post0
python-dotenv 1.1.1
python-json-logger 4.0.0
python-multipart 0.0.20
pytz 2025.2
PyYAML 6.0.3
pyzmq 27.1.0
ray 2.50.1
referencing 0.37.0
regex 2025.9.18
requests 2.32.5
rfc3986 1.5.0
rich 14.2.0
rich-toolkit 0.15.1
rpds-py 0.27.1
safetensors 0.6.2
scipy 1.16.2
sentencepiece 0.2.1
sentry-sdk 2.42.0
setuptools 80.9.0
shellingham 1.5.4
six 1.17.0
smmap 5.0.2
sniffio 1.3.1
starlette 0.46.2
sympy 1.13.1
tensordict 0.7.2
tiktoken 0.12.0
tokenizers 0.21.4
torch 2.6.0
torchaudio 2.6.0
torchdata 0.11.0
torchvision 0.21.0
tqdm 4.67.1
transformers 4.51.1
triton 3.2.0
typer 0.19.2
typing_extensions 4.15.0
typing-inspection 0.4.2
tzdata 2025.2
urllib3 2.5.0
uvicorn 0.38.0
uvloop 0.22.1
verl 0.2.0.dev0 /mnt/petrelfs/wangxiaoyang/yangzhou/my_project/VisionThink
vllm 0.8.5
wandb 0.22.2
watchfiles 1.1.1
websockets 15.0.1
wheel 0.45.1
wrapt 1.17.3
xformers 0.0.29.post2
xgrammar 0.1.18
xxhash 3.6.0
yarl 1.22.0
zipp 3.23.0
when I run the bash scripts,i got the error:
�[36m(WorkerDict pid=25427)�[0m INFO 10-21 15:17:02 [gpu_model_runner.py:1347] Model loading took 3.2152 GiB and 3.836757 seconds
�[36m(WorkerDict pid=25888)�[0m WARNING 10-21 15:16:57 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7f43a6e0c8d0>�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25888)�[0m INFO 10-21 15:16:58 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25888)�[0m INFO 10-21 15:16:58 [cuda.py:221] Using Flash Attention backend on V1 engine.�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25888)�[0m WARNING 10-21 15:16:58 [topk_topp_sampler.py:69] FlashInfer is not available. Falling back to the PyTorch-native implementation of top-p & top-k sampling. For the best performance, please install FlashInfer.�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25888)�[0m INFO 10-21 15:16:58 [gpu_model_runner.py:1329] Starting to load model Qwen/Qwen3-1.7B...�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25890)�[0m INFO 10-21 15:17:00 [weight_utils.py:265] Using model weights format ['*.safetensors']�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25889)�[0m INFO 10-21 15:17:05 [weight_utils.py:281] Time spent downloading weights for Qwen/Qwen3-1.7B: 0.568996 seconds
�[36m(WorkerDict pid=25427)�[0m INFO 10-21 15:17:13 [backends.py:420] Using cache directory: /mnt/petrelfs/wangxiaoyang/.cache/vllm/torch_compile_cache/d5db4b5d69/rank_0_0 for vLLM's torch.compile
�[36m(WorkerDict pid=25427)�[0m INFO 10-21 15:17:13 [backends.py:430] Dynamo bytecode transform time: 10.44 s
�[36m(WorkerDict pid=25889)�[0m INFO 10-21 15:17:07 [loader.py:458] Loading weights took 1.17 seconds�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25889)�[0m INFO 10-21 15:17:07 [gpu_model_runner.py:1347] Model loading took 3.2152 GiB and 8.628302 seconds�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25427)�[0m INFO 10-21 15:17:17 [backends.py:136] Cache the graph of shape None for later use
�[36m(WorkerDict pid=25889)�[0m INFO 10-21 15:17:17 [backends.py:420] Using cache directory: /mnt/petrelfs/wangxiaoyang/.cache/vllm/torch_compile_cache/d5db4b5d69/rank_0_0 for vLLM's torch.compile�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25889)�[0m INFO 10-21 15:17:17 [backends.py:430] Dynamo bytecode transform time: 10.55 s�[32m [repeated 7x across cluster]�[0m
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] EngineCore failed to start.
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] Traceback (most recent call last):
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 387, in run_engine_core
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] engine_core = EngineCoreProc(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 329, in init
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] super().init(vllm_config, executor_class, log_stats,
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 71, in init
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self._initialize_kv_caches(vllm_config)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/engine/core.py", line 129, in _initialize_kv_caches
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] available_gpu_memory = self.model_executor.determine_available_memory()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/executor/abstract.py", line 75, in determine_available_memory
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] output = self.collective_rpc("determine_available_memory")
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/executor/uniproc_executor.py", line 56, in collective_rpc
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] answer = run_method(self.driver_worker, method, args, kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/utils.py", line 2456, in run_method
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return func(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return func(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/worker/gpu_worker.py", line 183, in determine_available_memory
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.model_runner.profile_run()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1651, in profile_run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] hidden_states = self._dummy_run(self.max_num_tokens)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return func(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/v1/worker/gpu_model_runner.py", line 1497, in _dummy_run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] outputs = model(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self._call_impl(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return forward_call(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/model_executor/models/qwen3.py", line 299, in forward
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] hidden_states = self.model(input_ids, positions, intermediate_tensors,
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/decorators.py", line 238, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] output = self.compiled_callable(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 574, in _fn
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return fn(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 1380, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self._torchdynamo_orig_callable(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 547, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return _compile(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 986, in _compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] guarded_code = compile_inner(code, one_graph, hooks, transform)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 715, in compile_inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return _compile_inner(code, one_graph, hooks, transform)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_utils_internal.py", line 95, in wrapper_function
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return function(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 750, in _compile_inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] out_code = transform_code_object(code, transform)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1361, in transform_code_object
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] transformations(instructions, code_options)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 231, in _fn
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return fn(*args, **kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 662, in transform
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] tracer.run()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2868, in run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] super().run()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 1052, in run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] while self.step():
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 962, in step
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.dispatch_table[inst.opcode](self, inst)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3048, in RETURN_VALUE
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self._return(inst)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 3033, in _return
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.output.compile_subgraph(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1101, in compile_subgraph
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.compile_and_call_fx_graph(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1382, in compile_and_call_fx_graph
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn = self.call_user_compiler(gm)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1432, in call_user_compiler
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self._call_user_compiler(gm)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1483, in _call_user_compiler
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1462, in _call_user_compiler
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn = compiler_fn(gm, self.example_inputs())
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/dynamo/repro/after_dynamo.py", line 130, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_gm = compiler_fn(gm, example_inputs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/dynamo/repro/after_dynamo.py", line 130, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_gm = compiler_fn(gm, example_inputs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/init.py", line 2385, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self.compiler_fn(model, inputs, **self.kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/backends.py", line 459, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] PiecewiseCompileInterpreter(self.split_gm, submod_names_to_compile,
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/backends.py", line 249, in run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return super().run(*fake_args)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/fx/interpreter.py", line 167, in run
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.env[node] = self.run_node(node)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/fx/interpreter.py", line 230, in run_node
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return getattr(self, n.op)(n.target, args, kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/backends.py", line 265, in call_module
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiler_manager.compile(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/backends.py", line 125, in compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_graph, handle = self.compiler.compile(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/compiler_interface.py", line 318, in compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_graph = compile_fx(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1552, in compile_fx
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return compile_fx(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1863, in compile_fx
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return aot_autograd(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 83, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] cg = aot_module_simplified(gm, example_inputs, **self.kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 1155, in aot_module_simplified
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn = dispatch_and_compile()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 1131, in dispatch_and_compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn, _ = create_aot_dispatcher_function(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 580, in create_aot_dispatcher_function
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return _create_aot_dispatcher_function(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 830, in _create_aot_dispatcher_function
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn, fw_metadata = compiler_fn(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 203, in aot_dispatch_base
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fw = compiler(fw_module, updated_flat_args)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 489, in call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self.compiler_fn(gm, example_inputs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1741, in fw_compiler_base
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return inner_compile(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/contextlib.py", line 81, in inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return func(*args, **kwds)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/vllm/compilation/compiler_interface.py", line 229, in hijacked_compile_fx_inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] output = torch._inductor.compile_fx.compile_fx_inner(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 569, in compile_fx_inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return wrap_compiler_debug(_compile_fx_inner, compiler_name="inductor")(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_dynamo/repro/after_aot.py", line 102, in debug_wrapper
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] inner_compiled_fn = compiler_fn(gm, example_inputs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 685, in _compile_fx_inner
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] mb_compiled_graph = fx_codegen_and_compile(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1129, in fx_codegen_and_compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return scheme.codegen_and_compile(gm, example_inputs, inputs_to_check, graph_kwargs)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/compile_fx.py", line 1044, in codegen_and_compile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_fn = graph.compile_to_module().call
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/graph.py", line 2027, in compile_to_module
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] return self._compile_to_module()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/graph.py", line 2068, in _compile_to_module
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] mod = PyCodeCache.load_by_key_path(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/codecache.py", line 2759, in load_by_key_path
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] mod = _reload_python_module(key, path)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/runtime/compile_tasks.py", line 45, in _reload_python_module
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] exec(code, mod.dict, mod.dict)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/.cache/vllm/torch_compile_cache/d5db4b5d69/rank_0_0/inductor_cache/wi/cwig2bngj4tvov3mwyskdakscc6wudtjfq3ryocu6onlxrllz5l6.py", line 139, in
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] triton_poi_fused_mul_silu_1 = async_compile.triton('triton_poi_fused_mul_silu_1', '''
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/async_compile.py", line 213, in triton
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] kernel.precompile()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 293, in precompile
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] compiled_binary, launcher = self._precompile_config(
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 520, in _precompile_config
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] binary._init_handles()
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/triton/compiler/compiler.py", line 384, in _init_handles
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] self.run = driver.active.launcher_cls(self.src, self.metadata)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/triton/backends/nvidia/driver.py", line 440, in init
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] mod = compile_module_from_src(src, "__triton_launcher")
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "/mnt/petrelfs/wangxiaoyang/miniconda3/envs/visionthink-train/lib/python3.11/site-packages/triton/backends/nvidia/driver.py", line 62, in compile_module_from_src
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] mod = importlib.util.module_from_spec(spec)
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "", line 573, in module_from_spec
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "", line 1233, in create_module
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] File "", line 241, in _call_with_frames_removed
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] torch._dynamo.exc.BackendCompilerFailed: backend='<vllm.compilation.backends.VllmBackend object at 0x7f7ec8761a10>' raised:
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] ImportError: /mnt/petrelfs/wangxiaoyang/.cache/vllm/torch_compile_cache/d5db4b5d69/rank_0_0/triton_cache/229d4YYAkdh66Kgyl7jG-8awB8K1oU306NkqKGcFacU/__triton_launcher.so: cannot open shared object file: No such file or directory
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396]
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396]
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396]
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] You can suppress this exception and fall back to eager by setting:
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] import torch._dynamo
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396] torch._dynamo.config.suppress_errors = True
�[36m(WorkerDict pid=25887)�[0m ERROR 10-21 15:17:20 [core.py:396]
Question 2: Is this error caused by incorrect environmets?
Question 3:Can you give me some suggestions to solve it ?
Looking forward to your reply!