Reproducing Adapterfusion paper results

[timaurel]

Hello,

I would like to ask for help regarding reproducing the AdapterFusion results (table 1 in the paper)

This topic was briefly discussed here #112 but the answer was a bit inconclusive and since then, there have been a lot of updates to the repository.

Could the paper results be reproduced with 'run_glue.py' or would you advise writing a new script for this? Also, the script does not seem to produce evaluation metrics out of the box as running the script results in metrics like below:

***** train metrics *****
  epoch                    =        3.0
  train_loss               =     0.3172
  train_runtime            = 0:08:36.98
  train_samples            =       8551
  train_samples_per_second =      49.62
  train_steps_per_second   =      1.555
100% 131/131 [00:07<00:00, 16.80it/s]
***** eval metrics *****
  epoch                   =        3.0
  eval_runtime            = 0:00:07.91
  eval_samples            =       1043
  eval_samples_per_second =    131.728
  eval_steps_per_second   =     16.545

[hSterz]

Hey @timaurel , the run_glue.py script does not use Fusion modules. You would have to adapt the script to use the Fusion setup described in the paper. This mainly concerns adding and activating adapters (see here). To further help with your question it would be great if you could share some context: What arguments are you running the glue script with? Which results are you trying to reproduce?

[timaurel]

Hi @hSterz,

Thanks for you reply! I started looking into the repository a bit more and understand now what I have to do.

I am trying to reproduce the single task adapter results from the paper ("ST-A"/4th column in table 1). I have a script that downloads the data from Hugging Face for a given task, encodes it and trains an adapter on it. However, I consistently get worse performance than the results reported in the paper and I do not know why.

I am using the configuration mentioned in the paper (one output adapter, no multi-head adapter, reduction factor 2, learning rate 1e-4,etc.) but have not been able to reproduce the results for any of the datasets. I also tried different reduction factors (The paper mentions trying 2,16,64) but this does not help. I also wrote a dedicated notebook for each dataset, here is the notebook for the boolq dataset.
https://colab.research.google.com/drive/1Q3golpGFHWLBVfgCGv0jHpj-IwBh_far?usp=sharing

All the other notebooks (and my scripts) apply the same logic and only differ in the dataset, encoding and whether or not they use a classification head or a multiple choice head. Would you have ideas for why my performance is worse for all datasets? My results are significantly worse than the ones in the paper, often 10-15% lower. Thank you very much!