CONTRIBUTING.md

tfl-training-ml-control - developing the training

This repository was created from an aAI internal template

Overview of the features

When you create a new repository from the template, you will get the following features:

1. An example notebook, fully set up for presentation and participants. It contains a lot of documentation, so you can start by having a look at it.
2. A CI pipeline that will build a docker image and push it to our registry.
3. The source code is installable, you can run pip install -e ".[test]".
4. There are bash scripts for helping with various build steps in build_scripts. They are all documented, so feel free to call them with the -h flag. For a local setup, call bash build_scripts/install_presentation_requirements.sh
5. The notebooks are rendered to documentation and published as GH pages. They are also rendered to a jupyter-book. The rendering happens as part of the build_scripts/build_docs.sh script.
6. Data is downloaded with accsr and stored in the data directory. This happens on demand locally, and as part of the entrypoint in the docker container.

Setup

Accessing and pushing data

The configuration is already set up to work out of the box, you just need to set the env var TFL_STORAGE_ADMIN_SECRET. You can upload data by calling python scripts/upload_data.py.

Virtual environment based

Some requirements don't work well with pip, so you will need conda. Create a new conda environment, activate it and install dependencies with

bash build_scripts/install_presentation_requirements.sh 
pip install -e ".[test]"  

NOTE: Anecdotal evidence suggests that pip might actually work or that the conda-based installation can fail on some MacOS versions. If you run into issues, feel free to resolve them on the template repository.

Docker based

As the docker image is being created automatically and contains all requirements, you can simply use it for local development (e.g. using a docker interpreter in your IDE).

In summary, to get a running container you could do something like

docker pull europe-west3-docker.pkg.dev/tfl-prod-ea3b/tfl-docker/tfl-training-ml-control
docker run -it -p 8888:8888 --entrypoint bash \ 
-v $(pwd):/home/jovyan/tfl-training-ml-control \
docker.aai.sh/tl/trainings/tfl-training-ml-control  

NOTE: You will need to authenticate into the docker registry before pulling. This can be done with

docker login -u _json_key -p "$(cat <credentials>.json)" https://europe-west3-docker.pkg.dev 

Ask somebody for the credentials.

You can also build the image locally. Set the PARTICIPANT_BUCKET_READ_SECRET environment variable and run

docker build --build-arg PARTICIPANT_BUCKET_READ_SECRET=$PARTICIPANT_BUCKET_READ_SECRET -t tfl-training-ml-control .

For participants

Participants will start a jhub server based on the docker image built from the master branch. For them, everything should work out of the box.

Writing new content

Start by having a thorough read through nb_10_PrsentationTemplate.ipynb. Since IDEs don't support all the features of jupyter, it is recommended to view it through jupyter notebook.

For creating a new notebook, the easiest way is to copy the nb_20_CopyMe.ipynb notebook and to modify it. You should remember the following things:

1. Keep the naming consistent, i.e. nb_10_..., nb_20_..., etc.
2. Make sure to set up the slideshow metadata in the cells correctly.
3. You can remove cell input and output or remove entire cells from the rendered html using tags. Note that removed cells are not executed during the render, so if you want to remove something like an import cell, you can add the tags remove-input and remove-output (but not remove-cell).
4. You can declare cells as initialization cells, and you can hide them from the viewer, see the example notebook for details.
5. The tag hide-cell is ignored by nbocnvert but used by jupyter-book. It is useful for import cells. You can also consider using it for the hint-loading cells.
6. Extend notebooks/intro.md as you see fit. It is part of the jupyter-book.

Rendering and committing notebooks

Notebooks are a bit tricky. We don't want to execute them in CI, because that would take too long. We also don't want to commit the output, because it would make the repository too big. So we have to render them locally and commit the rendered html.

To render the notebooks, you can use build_scripts/build_docs.sh --execute. It will create files in docs/_build/html (with sphinx) as well as in notebooks/_build/html (with jupyter-book).

IMPORTANT: With the --execute option, the above script will also create files inside docs/_static. These files have to be committed! Don't worry about them blowing up the repository, they are stored in git lfs.

DEPRECATED: . If you are using the %load magic command for, this will collide with rendering. The current workaround is to first execute the cell containing %load before rendering the notebook. Note however, that the executed cells should not be committed! This means, for rendering and committing the html in docs/_static, you have to first execute the %load cells, then render the notebooks, and then revert the changes to the notebooks. The notebooks are validated before render and before the docker image is built, so you will get an error if you forget to do this.

Before the training

Make sure everything is committed as described above and pushed to the remote repository. Have a look at the GitHub pages: do the exercises pages and the jupyter-book look good? Ask the operations team to create a jupyterhub server and to include the docker image that was built in CI into the list.

Start an instance and check whether:

1. All notebooks can be executed.
2. All slideshows look good.

Note that the data is pulled in the background when you first open the training instance, so it might take a while. This is usually not a problem because in the first minutes of the training there is a presentation anyway.

Preparing the package

After the training is finished, you should distribute the code to the participants. We distribute a zip file containing source code, a read access key for the data, and the rendered html files.

To create the package, you should:

1. Replace the reference to the env var by the actual value in the secret entry in config.yml. IMPORTANT: only give out the read access key to the participants!
2. Execute all %load cells in the notebooks.
3. Render the docs as described above. Have a look at the rendered html files, both in docs/_build/html and notebooks/_build/html. If you are happy with the result, proceed.
4. Run build_scripts/prepare_zip_package.sh. It will create a zip file.
5. Send the package to the participants per e-mail.

Note: The final package includes documentation on the training's code. This docu is generated using sphinx. To use math notation in sphinx, make sure to adhere to their math syntax. For inline math see below. Further information can be found in their documentation on math support.

:math:`p(x) = N(mu, cov)`

Notes on CI Pipelines

The CI pipeline will build a docker image and push it to a registry in gcc, namely to this path. This image will be used in subsequent CI steps and for the training itself in jupyterhub. In order for the code to be in the participant's home directory after starting a jhub server, it is moved to it during the entrypoint (see the Dockerfile for details).

The notebooks will be executed as part of the build, rendered to documentation and published as gitlab pages (Note: currently not available). Currently, the theme we use in nbsphinx obliterates the custom styles that we use in the presentation, so the pages will not look as pretty as the presentation or the notebooks in jupyterhub. Still, they are useful for seeing whether notebooks look as you would want them to do (and whether they run through).

Docker builds with heavy requirements

You might run into problems when trying to build an image with heavy packages, e.g. pytorch. The filesystem changes can take too much RAM (16 GB was not enough on several occasions) and cause the build to fail.

In that case, you should create a separate file called requirements-light.txt and reference that one in the Dockerfile instead of the full requirements.txt. The participants will need to perform a local installation of the package anyway and can then install the heavier requirements into the container, without needing to build an image. It is not a great solution, but it is enough for now.

Structuring the code and testing

Source code should be in the src directory, tests belong inside tests (follow the same folder structure as src) and notebooks belong in notebooks.

Configuration and styles

There are two places for configuration: the rise.css file and the notebook's metadata. You can edit the latter either by opening the notebook as a raw document or by going to Edit->Edit Notebook Metadata in jupyter's interface. They already contain sensible, aAI-conforming defaults.

Do not use any of the other configuration options provided by rise! (unless you really know what you are doing).

Migrating previous trainings

It is not too difficult to migrate previous trainings to the structure in this template. Essentially, replace everything outside of src and notebooks by the new files, and adjust notebook paths (move images to notebooks/_static) and tags. You have to:

1. Replace all build scripts by the ones in this template.
2. Add the config.yml and config_local.yml files.
3. Add the upload/download_data.py scripts.
4. Add the non-notebook files inside notebooks
5. Replace the contents of docs by the ones in this template.
6. Remove anything pytest related inside notebooks
7. Move all images to notebooks/_static and adjust references in notebooks (including metadata) accordingly.
8. Add tags to the notebook cells as described above
9. Replace the contents of .github by the ones from thesan_output
10. Follow the instructions above to commit html files
11. Replace the Dockerfile and the entrypoint by the ones in this template.

If that sounds like too much, it might be easier to just create a new repository, copy over the notebooks and adjust paths and tags.