There was no speed improvement and gpu memory did not decrease.

[coder4nlp]

When I used VisionZip on qwen2.5-vl-7b, compared with the original implementation, I found that although the number of visual tokens decreased, the speed of inference did not increase and the amount of GPU memory did not decrease. Especially the GPU memory, it even increased more than before the compression. This was mainly caused by the following code.

if layer_num == len_blocks-1:
    hidden_states, logits, attn_key = blk(hidden_states, cu_seqlens=cu_seqlens_now, position_embeddings=position_embeddings, return_logits=True)

[MQN-80]

I believe the main reason is that FlashAttention doesn’t yield the full attention matrix, so they fall back to the original QKV computation to obtain it. Doing so not only fails to shorten inference time but also drastically increases memory usage...

[Yangsenqiao]

Thanks for your interest in our VisionZip.

Here are my thoughts on this issue:

1. Since we need to compute the attention weights, and Flash Attention does not directly provide them, it is inevitable that we need to recompute attention in the vision encoder or resort to methods other than Flash Attention.
2. You may try using VLLM and run with a maximized batch size to see if it improves the overall runtime on the full dataset.

Looking forward to further discussion with you.

[coder4nlp]

Thanks for your interest in our VisionZip.

Here are my thoughts on this issue:

1. Since we need to compute the attention weights, and Flash Attention does not directly provide them, it is inevitable that we need to recompute attention in the vision encoder or resort to methods other than Flash Attention.
2. You may try using VLLM and run with a maximized batch size to see if it improves the overall runtime on the full dataset.

Looking forward to further discussion with you.

@Yangsenqiao The purpose of visual token compression is to reduce the amount of memory and speed up the inference time. However, when dealing with high-resolution images, the required attention scores are extremely costly, which is contrary to the original intention of the paper.So sad.

[Yangsenqiao]

The current VisionZip port was mainly proposed for the LLaVA family. Qwen 2.5‑VL was released after our paper, so the integration is still experimental. We believe there are many issues yet to be addressed, and VisionZip is by no means the final solution for building efficient VLMs.

That said, there's still a long road ahead — but we’re excited about the rich space for exploration in this area 🌱. We're also looking forward to seeing newer and more efficient methods emerge!

By the way, we recently released a project VisionThink, which is one such exploration toward efficient reasoning VLMs for Qwen2.5‑VL 🚀.

[YirongWho]

A possible design could be to disable Flash Attention exclusively for the penultimate layer of the CLIP encoder, while enabling it for all the earlier layers. What do you think? @Yangsenqiao