Global-local Feature Alignment

[1024er]

Hi,
Have you tried the InfoNCE loss in Global-local Feature Alignment ?

[CLS] and [MSK] in the same sentence constitute positive pairs
[CLS] and [MSK] in different sentence constitute negative pairs

[LiJunnan1992]

Hi, thanks for your question. We have tried InfoNCE, but it does not work as well as the squared distance. Our hypothesis is that InfoNCE is too "simple" and does not regularize the model well enough.