torch.compile and flashattention implementation in sonar

[nitishpandey04]

Is torch.compile and flashattention already used in sonar implementation? if not, how to use it to make it faster?

[sl5035]

have you figured it out?

[nitishpandey04]

Not using fairseq2 but there's an implementation of Sonar given in huggingface by one of the authors of the paper. There you can make any modifications you want according to the transformers library API.

https://huggingface.co/cointegrated/SONAR_200_text_encoder