Incompatibility of Newly Released Large Parameter Models (650M & 111M) with Existing Code

[Gengsheng-Li]

Dear Authors,

Hello! Thank you very much for open-sourcing this incredibly inspiring project. While using the models provided, we noticed that the newly released 650M and 111M models seem to be incompatible with the existing code. The issue we encountered is as follows:

The error occurs in the BrainLMDecoder class's forward function within the brainlm_mae/modeling_brainlm.py file:

File "/data0/user/gsli/BrainLM/brainlm_mae/modeling_brainlm.py", line 716, in forward
    decoder_outputs = self.decoder(latent, xyz_vectors, ids_restore)
File "/data0/user/gsli/.conda/envs/brainlm/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1130, in _call_impl
    return forward_call(*input, **kwargs)
File "/data0/user/gsli/BrainLM/brainlm_mae/modeling_brainlm.py", line 508, in forward
    x_ = x_ + xyz_projection
RuntimeError: The size of tensor a (512) must match the size of tensor b (1280) at non-singleton dimension 3

This error did not occur when using the old 13M model. Is there an updated version of the code compatible with the new models? Or could you provide some guidance on how to resolve this issue?

Once again, thank you for your excellent work and for sharing it with the community!

Looking forward to your response.

Best regards,
Gengsheng Li

[matteomattersberger]

Dear Gengsheng Li,

Let me preface this by stating I have no solution for your query. Quite the contrary, I am currently faced with similar issues - while the code (most of it at least, some parts need adjustment) seems to work with the old 13M model, the very architecture of the newer versions are seemingly different, making the existing code incompatible with it.

I am only answering - as you have posed this question quite a while ago - to inquire whether you happen to have had any success in the meanwhile in resolving the issue, and if so, if you would be willing to share your solutions.

Many thanks and all the best!

[Gengsheng-Li]

Dear Gengsheng Li,

Let me preface this by stating I have no solution for your query. Quite the contrary, I am currently faced with similar issues - while the code (most of it at least, some parts need adjustment) seems to work with the old 13M model, the very architecture of the newer versions are seemingly different, making the existing code incompatible with it.

I am only answering - as you have posed this question quite a while ago - to inquire whether you happen to have had any success in the meanwhile in resolving the issue, and if so, if you would be willing to share your solutions.

Many thanks and all the best!

For using the larger model with a size of 650M or 111M, you need to use a set of different model-building files recently published under BrainLM/brainlm_mae/, which should be modeling_vit_mae_with_padding.py and replace_vitmae_attn_with_flash_attn.py. :-D

This solution works for me.

[Gengsheng-Li]

Dear Gengsheng Li,

Let me preface this by stating I have no solution for your query. Quite the contrary, I am currently faced with similar issues - while the code (most of it at least, some parts need adjustment) seems to work with the old 13M model, the very architecture of the newer versions are seemingly different, making the existing code incompatible with it.

I am only answering - as you have posed this question quite a while ago - to inquire whether you happen to have had any success in the meanwhile in resolving the issue, and if so, if you would be willing to share your solutions.

Many thanks and all the best!

It happens that I was also a student of UCD, haha. But, I have graduated now.

[matteomattersberger]

Thank you for the tip. For adjusting the data from shape (424, 200), you simply pad it to shape 432x432 using the AutoImageProcessor and repeat the channel three times, have I understood that correctly?

[Gengsheng-Li]

Thank you for the tip. For adjusting the data from shape (424, 200), you simply pad it to shape 432x432 using the AutoImageProcessor and repeat the channel three times, have I understood that correctly?

Yes, the code finally will automatically mean the features across 3 channels to recover the original shape, so it doesn't matter.

[Gengsheng-Li]

Thank you for the tip. For adjusting the data from shape (424, 200), you simply pad it to shape 432x432 using the AutoImageProcessor and repeat the channel three times, have I understood that correctly?

In fact, all channels contain the same information. I guess the reason for constructing data like this is just to fit some input requirements of the called framework.

[matteomattersberger]

Great, thank you for confirming that. If I may ask you one more question, have you manually min-max normalized your data or applied any other preprocessing steps to keep the data within the boundaries (0,1)? The Preprocessing employed by the authors does not confine the fmri data to the range 0,1, however, this is the range the AutoImageProcessor seems to require to process the fmri data to the proper pixel format.