About training KANs on embeddings of pre-trained models

[hoangthangta]

Hi there,

This is for everyone who cares about using KANs for text classification. Throughout our work, we have some notes which may need to be confirmed and discussed:

1. All KANs face convergence issues during training when they depend exclusively on embeddings from pre-trained models without further fine-tuning. In our experiments, we tried various techniques such as data normalization, dropout, and adjusting the learning rate, but none improved the training and validation accuracy values. The highest accuracy we achieved with Efficient KAN on MRPC was 0.74 %, much lower than other popular methods. However, the training time is significantly faster, and the saved model has a small capacity.
2. To solve 1, we insert a pre-trained model like BERT into KANs and fine-turn it, which works and archives a competitive performance. However, KANs take longer to train than other networks, such as Transformer Classifiers (with dropout and linear modules) and Transformer MLP.

That is what we know so far, and we will continue to update this discussion.