README.md

       

Quickly compare PyTorch models in a local, lightweight registry

---

tvault is designed to help academic researchers iterate their models faster without logging overheads.

Many of the academic researchers we encounter simply want to get going with minimum setup and configurations. This often means using local codebase as opposed to integrating tools like VESSL Experiments and Weights & Biases to git-committed codes.

tvault is "git diff for machine learning" — a simple, lightweight framework for quickly tracking and comparing ML experiments in a local model registry.

• Track and version models locally with tvault.log_all()
• Get a birds-eye differences of two experiments with tvault --diff_flag

Get started with pip install:

pip install tvault

For those who are already using VESSL Python SDK and CLI,

pip install "vessl[tvault]"

Follow our guide below with our MNIST example code.

---

Getting started with tvault.log()

Insert tvault.log() in your code's training loop with the metrics you want to track as tags.

tvault/assets/mnist-train.py

Lines 145 to 146 in 4bebfe4

	tags = {"language": "pytorch", "size": "0.5x", "learning_rate": learning_rate}
	tvault.log_all(model, tags=tags, result=acc.item(), optimizer=optimizer)

tvault.log() will automatically create the following:

• A folder model_log under your current directory - a tracking dashboard or model registry for your code
• A unique hash for the model and model ID for each training run with the key metrics

Look up experiments with tvault --find_flag

tvault's find_flag option allows you to look up different experiments with a simple CLI command. find_flag offers three different ways of exploring results:

1. Search by hash — The command below shows all experiments with the hash value of 2ba4adf.

tvault --find_flag --condition hash --hash 2ba4adf

2. Search by result — The command below shows all experiments with the result between 50 and 100.

tvault --find_flag --condition result --min 90 --max 100

3. Search by tags — The command below shows all experiments tagged as 0.5x.

tvault --find_flag --condition tag --tag_type size --tag 0.5x

Compare models with tvault --diff_flag

tvault's diff_flag option allows you to look up the difference of two models by specifying the model hash and index. tvault automatically detects and displays the changes in functions while removing git diffs that are not related to the model.

This is useful when you have a baseline model that you want to iterate with different hyperparameters and higher-level architectures without digging through your code line-by-line. tvault can also get the diff related to the optimizers.

tvault --diff_flag --sha1 f407ed0 --index1 0 --sha2 737b47a --index2 0

Issues, feature requests, and questions

We are excited to hear your feedback!

• For issues and feature requests, please open a GitHub issue.
• For questions, please use GitHub Discussions.
• For general discussions, join our community Slack.

VESSL for Academics

Our free academic plan is dedicated to help graduate students set up a modern ML research workflow with zero maintenance overheads. Learn more about VESSL's full-fledged, fully-managed job scheduler and experiment tracking.