train language on t5

[crescent73]

I am trying to train a language adapter on the t5(encoder-decoder) model.
I followed the pre-training script of t5 and translate it into pytorch version . then I add a language adapter and start training.
training details are as follows:
adapter config PfeifferInvConfig(reduction_factor=2), step 25,000, lr 1e4, warmup_ratio 0.1, train batch 24.
The final model loss is about 2.2, and the mlm accuracy is about 64%.
However, the language adapter I trained didn't work as well ,and even worse on downstream.
I don't know whether there is something wrong with the language adapter I trained, which leads to poor results in the downstream application. Could you please give me some suggestions? @JoPfeiff

[adapter-hub-bert]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[crescent73]

您好，我已收到您的来信!Hello, I've received your letter!

[adapter-hub-bert]

This issue has been automatically marked as stale because it has been without activity for 90 days. This issue will be closed in 14 days unless you comment or remove the stale label.

[adapter-hub-bert]

This issue was closed because it was stale for 14 days without any activity.