-
Notifications
You must be signed in to change notification settings - Fork 938
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[fix] VisualBERT returns attention tuple #1036 #1052
base: main
Are you sure you want to change the base?
[fix] VisualBERT returns attention tuple #1036 #1052
Conversation
PROBLEM: The default value of output_attentions in forward( ) call of BertEncoderJit (in mmf/modules/hf_layers.py) is set as False. So even if the user/developer specifies output_attentions = True in config; its value is taken as default False and thus VisualBERT returns an empty tuple for attentions. FIX: Set output_attentions as None in BertEncoderJit's forward( ) definition, and update output_attentions to self.output_attentions if it is not passed as an argument (i.e it is None). Therefore, now output_attentions will take the value of self.output_attentions (which was initialized using config during instantiation of BertEncoderJit class) Same problem and same fix for output_hidden_states as well. Tested locally.
Hi @abhinav-bohra! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks! |
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for making this changes and contributing to MMF. Can you write a simple test to verify that this actually works. Test should go in tests/models/test_visual_bert.py
.
Thanks for your feedback @apsdehal . I will add the tests. I had one doubt - there can be 3 places where the user can set output_attention, namely;
For example:
Should the priority be: forward > model > config ? Or should it be an 'OR' of all the three cases; i.e. setting it anywhere as True will return the attention_list ? |
PROBLEM: The default value of output_attentions in forward( ) call of BertEncoderJit (in mmf/modules/hf_layers.py) is set as False. So even if the user/developer specifies
output_attentions = True
in config; its value is taken as default False and thus VisualBERT returns an empty tuple for attentions.FIX: Set
output_attentions as None
in BertEncoderJit's forward( ) definition, and update output_attentions to self.output_attentions if it is not passed as an argument (i.e it is None). Therefore, output_attentions will now take the value of self.output_attentions (which was initialized using config during instantiation of BertEncoderJit class)The issue with
output_hidden_states
was the same, and it was fixed in a similar way.Tested locally.