Fine tuning LLM

[manisnesan]

https://lightning.ai/pages/community/finetuning-falcon-efficiently/

[manisnesan]

https://lightning.ai/pages/blog/scaling-large-language-models-with-pytorch-lightning/

[manisnesan]

https://lightning.ai/pages/community/lora-insights/

[manisnesan]

Answer.ai post - You can train 70B param model using FSDP and QLora

• scale resource-efficient QLoRA training across inexpensive gaming GPUs
  • will help bring more attention to the problem of bringing down the cost of model training.
• It’s in everyone’s interest to make AI more accessible – and to enable more people to not only consume, but also build, valuable models.

[manisnesan]

LoRA - Low rank adapters.
They are basically small matrices. Keeping the rest of the model as constant, only train these small matrices

Intent is everybody need to contribute to the creation of models

LoRA doesn’t train the whole large language model at all, but instead adds “adaptors”, which are very small matrices (generally smaller than 1% of the full model) that are trained, whilst keeping the rest of the model constant

Keeping the base model as quantized ( frozen during training) keep the adapters unquantized

Tim realized that LoRA can be combined with quantization: use a quantized base model, which is not changed at all by the training, and add trainable LoRA adaptors that are not quantized. This combination is QLoRA

[manisnesan]

PEFT

Parameter Efficient Fine Tuning- PEFT approaches enable you to get performance comparable to full fine-tuning while only having a small number of trainable parameters.

[manisnesan]

Fine tune minimal expample using QLORA - Colab

[manisnesan]

Fine tune using Unsloth with Colab
Examples

Very few lines of code + GPU poor friendly + Good performance

X post

[manisnesan]

Fine tune your first LLM using torch tune

torch tune

Reference: https://github.com/pytorch/torchtune

Source : Andrej tweet

[manisnesan]

Fine-tune Llama 3 with ORPO

• introduced the ORPO algorithm and explained how it unifies the SFT and preference alignment stages into a single process.
• used TRL to fine-tune a Llama 3 8B model on a custom preference dataset
• final model shows encouraging results and highlights ORPO's potential as a new fine-tuning paradigm.

Source : Maxime labonne post & another post

[manisnesan]

ORPO slides

[manisnesan]

Fine tune a gpt2 model for spam classification

https://github.com/rasbt/LLMs-from-scratch/blob/main/ch06/01_main-chapter-code/ch06.ipynb

[manisnesan]

fine tune with axolotl

• Fine tune with smaller sample
• Fine tune with full dataset

[manisnesan]

https://lucasvw.github.io/posts/19_llm_fine_tuning/

[manisnesan]

Extended Guide: Instruction-tune Llama 2 - https://www.philschmid.de/instruction-tune-llama-2

Toc
1. Define the use case and create a prompt template for instructions
2. Create an instruction dataset
3. Instruction-tune Llama 2 using trl and the SFTTrainer
Flash Attention
4. Test Model and run Inference