Problem Statement: In most real-world applications, labelled data is scarce. Suppose you are given the Fashion-MNIST dataset (https://github.com/zalandoresearch/fashion-mnist), but without any labels in the training set. The labels are held in a database, which you may query to reveal the label of any particular image it contains. Your task is to build a classifier to >90% accuracy on the test set, using the smallest number of queries to this database.
Approach: Use a variational autoencoder for unsupervised training on the full unlabeled dataset. Then train the classifier on the encodings of the autoencoder.
The encoded features:
The images after reconstruction through the variational autoencoder:
This project demonstrates
Convolutional variational autoencoderin PytorchVS-Code dev containerwith CUDA support- Pytorch-Lightning for training. It would be possible to use vanilla Pytorch, however, if there exists a framework for it, it's better to use it.
- Torchmetrics for calculating accuracy and f1-scores.
- Hydra for experiment management. All experiments are named according to the deviation from the default configuration.
- Tensorboard for advanced visualization of the encoded labels, and training statistics.
Scalarsfor training progressImagesfor visualizing the reconstructed images from the autoencoderProjectorfor visualizing the embeddings
- Ax for hyperparameter optimization
- flake8 for linting, black for formatting, isort for import sorting.
Architecture is inspired by RDenseCNN and SAM Optimizer.
Side Note: Usually, I use poetry or conda as a package manager. However, the NVIDIA-Docker container comes with an existing environment, that suffers from a InvalidVersionSpec error. Conda is also incredible slow in this case, therefore I am using pip install to add missing dependencies to the Docker Image.
(base) vscode@54a0b6598fe8:/workspaces/efficient-mnist-learner$ conda env export
InvalidVersionSpec: Invalid version '1.10.0+cu113<1.11.0': invalid character(s)| Number of training labels | Accuarcy on validation |
|---|---|
| 1000 | 84.74% |
| 2000 | 86.25% |
| 5000 | 89.37% |
| 7500 | 90.42% |
| 10000 | 91.03% |
| 20000 | 91.65% |
| 30000 | 93.04% |
| 60000 | 93.56% |
The auto-encoder results in a boost of 2-3% over a model which has not been pre-trained on the whole dataset.
Open the project in the VS-Code dev container. Select the base conda environment that ships with the container. This is tested on CUDA Version 11.4, Ubuntu 20.04 LTS.
If you don't want to use VS-Code, you can run it in docker as well.
cp requirements.txt .devcontainer/
docker build .devcontainer/ -t devcontainer
docker run --rm -it -v $(pwd):/workspace devcontainer bashYou can run an experiment using the command line.
python -m emlParameters can be overwritten the following way:
python -m eml variational_sigma=0.001 unsupervised_epochs=10 classifier_epochs=10More details can be found on the Hydra documentation. All configuration options are described in the Config.py file.