On the Convergence of Attention Positions in SigLIP Across Different Images: A Contrast with CLIP

[HAOobserver]

In your paper, the figure 11 "Visualization of Redundancy in the SigLIP Model" shows the visualization of the attention maps of SigLIP. Why do the attentions of SigLIP roughly focus on the same positions for different images, while this problem does not occur in CLIP?

[HAOobserver]

like this

[Yangsenqiao]

To be honest, I’m also curious about this 🤔. Although the high-attention regions aren’t exactly the same, they do look very similar. My guess is that this might be related to the pretraining loss function used in SigLIP—but I’m not entirely sure. If anyone has insights, feel free to share! I'm really interested in how these foundation models behave 🔍