Directory for pre-trained and fine-tuned models.
- convert.py converts the original weights of the volta models to the transformers_volta models.
- batch_convert.sh is a batch version of convert.py to make the five V&L models.
- test_obj36.tsv contains some image features extracted by the original BUTD detector. Those features are used to make the default image features, features used when no image input is provided, for the V&L models.
- pretrained is a directory to place the pre-trained models.
- finetuned is a directory to output the fine-tuning results.
We provide two methods to prepared the weights: conversion and download.
The created models can be used from the transfomers_volta by specifing their path.
We describe the ctrl_vilbert case as an example.
-
download the original weight from the link ViLBERT (CTRL) in e-bug/volta/MODELS.md into the download/volta_weights directory, with the name ctrl_vilbert.
-
run convert.py from the repository root directory.
python convert.py --config download/volta_config/ctrl_vilbert_base.json --weight download/volta_weights/ctrl_vilbert
This will make a ctrl_vilbert (directory) in vl_models/pretrained.
We prepared batch_convert.sh to make the five models used our experiments (ctrl_visual_bert, ctrl_uniter, ctrl_vl_bert, ctrl_lxmert and ctrl_vilbert). Run this batch script from the repository root directory after downloading and putting the corresponding weights.
Alternatively, you can download model files we made.
Here is the url list.
To use those models, after downloading, unzip them and put the unzipped model directories into vl_models/pretrained.
We made those model files from the e-bug/volta work. If you use these weights, do not forget to cite their work appropriately.
Controlled models:
Reinitialized models (initialized randomly and transferred some weights from BERT-base-uncased):