Read the paper here.
To reproduce the experiments, complete the following steps:
Create a virtual environment and run the following:
pip install -r requirements.txt
to install all necessary requirements.
Run the following command to complete data pre-processing and bring all datasets to the BEIR format:
python3 data_preprocessing.py
For our experiments, we use the following four publicly available IR benchmarks: (i) HotpotQA and (ii) Natural Questions from the BEIR collection, as well as the (iii) Political Science and (iv) Computer Science collections from the Multi-Domain Benchmark. We bring the last two collections to the BEIR format for our experiments.
Update the parameters in the pipeline.sh file to the desired ones and execute the script.
For example, the following command will train sbmoe_ALL on TinyBERT using the HotpotQA dataset by employing 6 experts:
python3 1_train_new_moe.py model=tinybert dataset=hotpotqa testing=hotpotqa model.adapters.use_adapters=True model.adapters.num_experts=6 model.init.specialized_mode=sbmoe_all
_*Note: When use_adapters=True, only two modes are allowed; (i) 'sbmoe_all' and (ii) 'sbmoe_top1'.
3D t-SNE representations of the same query and its top 1000 documents, on the left embedded by the original DRM and on the right by our model, for all seven benchmarks.
