error when using in llava-1.6-vicuna-7b

[Sugar929]

Great jobs!

I follow the setting in your paper:

model = visionzip(model, dominant=540, contextual=100)

lmms-eval throws an error,

File "VisionZip/visionzip/clip_encoder.py", line 80, in forward
contextual_tokens = target_hidden + aggregated_hidden
RuntimeError: The size of tensor a (37) must match the size of tensor b (100) at non-singleton dimension 1
2025-07-23 00:06:03.869 | ERROR    | __main__:cli_evaluate:357 - Error during evaluation: The size of tensor a (37) must match the size of tensor b (100) at non-singleton dimension 1. Please set `--verbosity=DEBUG` to get more information.

[Yangsenqiao]

Hi @Sugar929 👋,

Thanks for your interest in our work! In my opinion, the error is caused by setting the total token count to 540 + 100 = 640, which exceeds the number of raw visual tokens (576). I recommend setting the values to 54 and 10, respectively.

[Sugar929]

@Yangsenqiao thank u~ in llava-1.5, the number of visual token is 576, I'm trying to eval llava-1.6, which token number is 2880.

dominant=540, contextual=100 this setting is in your appendix.

To adjust to version 1.6, is there any other hyperparameters should I change?

[Yangsenqiao]

Hi @Sugar929 👋,

It might not work by directly using the same setting, since prepare_inputs_labels_for_multimodal differs between LLaVA 1.5 and 1.6. I’d recommend porting the core logic over instead, as shown in VisionZip/visionzip/clip_encoder.py

If you have any further questions while implementing it, feel free to email me or continue the discussion here.

[XinweiZhang1998]

I have the same question. Can you also share the code on LLaVA 1.6?