lora/qlora training w/wo unsloth on NVIDIA L4 (~20GB VRAM usage)

[ajaymin28]

Pull Request Summary #12

issues addressed

• L4 GPU with ~20GB ram usage batch=16 Finetuning with GPU #9
• Added LoRA and QLoRA support [Add] LoRA support #10 Add QLoRA support file #13
• Save best weights based on validation loss Improved Training Loop: Add Validation and Best Checkpoint Saving #23

Major updates

• The files config.py, create_dataset.py, and utils.py (now renamed to utilities.py) have been moved to the utils directory to keep the project root organized and clean.

Summary

• Refactored and modularized the codebase to support extensive configuration(OmegaConf) and future extensions.
• Enhanced argument parsing to convert dataclass configurations to CLI arguments: dataclass → CLI args → final config.
• Implemented model checkpointing based on best validation loss and tested Weights & Biases (wandb) logging.
• Added support for pushing the best model checkpoint to the Hugging Face Hub.
• Find wandb report here: https://api.wandb.ai/links/ajaymin28-university-of-central-florida/4kw0i5o2
• Find Weights here: https://huggingface.co/ajaymin28/Gemma3_ObjeDet
• log file: unsloth_log_gemma3.log

---

What Works

• Training and validation using Unsloth, qlora vanilla method.
• Local saving of model checkpoints on best validation loss.
• Loading models from local directories.
• Automatically pushing the best model to Hugging Face Hub.

---

Common errors and resolutions

While Dataset loading if you get an error "Invalid pattern: '**' can only be an entire path component"
Refer this: https://stackoverflow.com/questions/77671277/valueerror-invalid-pattern-can-only-be-an-entire-path-component

AttributeError: 'Gemma3ModelOutputWithPast' object has no attribute 'loss': unslothai/unsloth#2656

---

Not Yet Tested

• The saved model checkpoint has not been thoroughly tested for inference yet. This is a high-priority task.

---

Packages

except colab:

uv pip install -r requirements.txt
uv pip install unsloth==2025.5.7 unsloth-zoo==2025.5.8 omegaconf

if you are on colab, refer this:

https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Gemma3_(4B).ipynb

---

commands used for training with unsloth on L4

In train.py, uncomment unsloth imports then run below command

python train.py --dtype=bfloat16 --finetune_method=qlora --batch_size=16 --use_unsloth=True --epochs=1 --max_step_to_train=1000 --validate_steps_freq=100

commands used for training without unsloth on L4

In train.py, comment unsloth imports, make sure FastModel = None, then run below command

python train.py --dtype=bfloat16 --finetune_method=qlora --batch_size=6 --use_unsloth=False --epochs=1 --max_step_to_train=1000 --validate_steps_freq=100

loss graphs

TODO

1. Test the trained model, publish results.
2. Train other methods, vanilla lora, vanilla qlora and test those models as well.
3. clean qlora and lora config and test for all configs (rslora, dora etc.).

This is my first contribution to GitHub, and I welcome any feedback or suggestions. I’ve aimed to keep the code simple, well-structured, and easily extendable for new modules, while following best practices throughout.

[ajaymin28]

Prediction Code and Outputs for Unsloth QLoRA

You can find the working inference code for Unsloth QLoRA in this branch:
https://github.com/ajaymin28/gemma3-object-detection/tree/predeval

The associated model is available here:
https://huggingface.co/ajaymin28/Gemma3_ObjeDet

Note:
The current predictions are not optimal, as the model was trained for only 100 steps.

For evaluation, I’ve provided a sample output pickle file:
outputs/infer_data_unsloth_qlora.pkl

The pickle file contains a list of dictionaries for the test set, with the following fields:

{
  "image_id": image_id,
  "width": width,
  "height": height,
  "output_text": output_text  # predicted text from the model
}

[ajaymin28]

Closing this for now, I don't think anyone needs this. Let me know otherwise.