Fine-tuning large language models like LLaMA has transformed the way we adapt pre-trained models for specialized tasks. This repository focuses on parameter-efficient fine-tuning techniques such as LoRA and QLoRA to adapt the LLaMA2-7B model to Indian legal text datasets.
You are tasked with fine-tuning the LLaMA2-7B model on a dataset related to Indian laws to make it capable of generating context-aware legal insights. The challenge is to leverage advanced fine-tuning techniques like LoRA/QLoRA to optimize the training process while keeping computational requirements minimal. Demonstrate your skills in model tuning and deployment!
-
Refer to articles, research papers, and official documentation for guidance on techniques and best practices.
-
Do not alter any pre-written code or comments.
-
Write code only in the provided space and document your steps with comments for better understanding.
-
Use Google Colab or similar GPU-enabled environments for training and testing the model.
-
Help
-
For any queries or support, feel free to reach out via email at iib2023013@iiita.ac.in or iit2023153@iiita.ac.in or join the discussion on the project’s Discord server.
-
Contributions are welcome! Follow these steps:
-
Fork this repository and clone it to your local device.
-
Work on individual tasks in a separate branch.
-
Push your updates to the forked repo and create a Pull Request (PR).
-
Your PR will be reviewed, and upon approval, merged into the main repository.
-
Dataset: Indian Law Dataset (https://huggingface.co/datasets/jizzu/llama2_indian_law_v2)
-
Parameter-Efficient Fine-Tuning: LoRA Paper (https://arxiv.org/pdf/1902.00751)
-
Hugging Face Transformers Documentation: Link(https://huggingface.co/docs/transformers/index)