Training Transformer on Small Dataset

[JakubSchwenkbeck]

Overview

This PR trains the Transformer model on a small dataset (50-100 words) to validate the forward and backward passes, loss function, and optimization. The goal is to observe the initial learning behavior and check for overfitting.

Dataset

A small story is used, tokenized into sequences:

Input:
"Once upon a time, in a land far away, there was a small village."

Target:
"The villagers were known for their kindness and generosity."

Training Process

• Model: Transformer
• Loss Function: Cross-entropy loss
• Optimizer: Adam with gradient clipping
• Metrics: Loss and accuracy logged

Expected Outcome

• The model should overfit due to the small dataset.
• Loss should decrease, and accuracy should increase as training progresses.

Conclusion

This small-scale training helps verify the core functionality before scaling to larger datasets.