Question about the token merge for QwenVL

[nzomi]

Hello authors, I noticed in your code the line

attn_mean = attn_mean.view(attn_mean.shape[0] // 4, -1).mean(dim=-1)

This operation seems to merge every 4 consecutive rows into one token. However, in Qwen the token merging is usually performed as a 2×2 spatial merge of neighboring tokens. Could you clarify if this implementation is intentional, or should it be a 2×2 merge instead of row-wise grouping？

[nzomi]

I checked the code, especially the get_window_index function, and now I understand why you can do this. My question is: is your way of computing attn_mean (using view and mean) actually different from simply taking the original attention weight (s, s), averaging along one dimension to get (s,), then reshaping it into (h, w) and applying a 2×2 pooling?

[nzomi]

The attention used here is window attention, so each token only attends to its local window. The view and mean operations are applied after local attention, not on global attention. If I were using a standard ViT with full global attention, then I could simply reshape and apply pooling.