This repository contains installation scripts and environment definitions for conda and texlive, for standardising environments across the MalariaGEN resource centre team.
This repo is intended to be used as a git submodule within another repo. This typically involves three distinct steps:
-
Add the malariagen/mgenv repo as a submodule to your repo. This only needs to be done once per repo. Note that someone else might already have done this for your repo of interest (the repo will include a
mgenvsub-directory if that is the case). -
Run the install script for each local clone of the repo. This needs to be done for each local clone you have, for example you might need to do this separately for one clone on your local machine and a second clone on a server.
-
Update the mgenv submodule. This needs to be done every time you know there have been changes made to the mgenv repo, and you want to move the submodule forward within your working repo.
The following sub-sections give commands for each of the above.
It is recommended that you first create an issue for doing this within your repo with a title such as "Add mgenv". Then create a new branch, add mgenv submodule, and create a pull request. It is also recommend that you add 'deps' to the .gitignore file for the repo, so that once the install command has been run (see next section), the deps directory this is installed into will not come under git tracking. For example, to use this within the malariagen/vector-ops repo, assuming your new issue is number 10:
cd /path/to/local/clone/of/vector-ops
git checkout -b 10_add_mgenv
git push -u origin 10_add_mgenv
git submodule add git@github.com:malariagen/mgenv.git
git add --all
git commit -m 'add mgenv submodule'
git push
At this point create a pull request for your new branch, and then pull the changes into master, deleting the new branch.
If you don't already have a local clone of the repo, create one.
Note you need to use --recursive in order to pull in mgenv
code. Example code for vector-ops repo:
git clone --recursive git@github.com:malariagen/vector-ops.git
To run the conda installation script, do:
cd /path/to/local/clone/of/vector-ops
./mgenv/install-conda.sh
Once conda is installed, you can activate the environment with:
source mgenv/env.sh
Once activated, you can run a jupyter notebook on a local machine with:
jupyter notebook
Or if you want to run a jupyter notebook on a server that you will connect to from your local machine, you could run a command such as:
jupyter notebook --no-browser --ip=0.0.0.0 --port <port number>
If you know there have been changes made to the mgenv repo, and you want to move the submodule forward within your working repo (e.g., vector-ops), do:
cd /path/to/local/clone/of/vector-ops
git checkout -b update-mgenv
cd mgenv
git fetch
git checkout v2.1.0
cd ..
git add mgenv
git commit -m 'update mgenv submodule'
git push -u origin update-mgenv
This will create a branch called 'update-mgenv' with the mgenv submodule moved forward to the "v2.1.0" tag. You can then get that into master via a PR.
Once merged, don't forget to update your conda environment on all machines, e.g.:
cd /path/to/local/clone/of/vector-ops
./mgenv/install-conda.sh
If you want to propose changes to this repository, e.g., add or update a package, please submit a pull request. The following commands show a typical workflow for doing this.
If you don't already have a local clone of the mgenv repo, fork the repo to your user account, then clone:
git clone git@github.com:yourusername/mgenv.git
Update your local master branch:
cd /path/to/local/clone/of/mgenv
git checkout master
git pull
Create a new issue at https://github.com/malariagen/mgenv/issues describing the changes you propose to make.
Create a new branch for the work you want to do, naming the branch using the issue number and a descriptive slug, e.g.:
git checkout -b 10-add-foo-package
git push -u origin 10-add-foo-package
Make changes to requirements-conda.txt or requirements-pypi.txt in a text editor.
Delete the pinned environment files:
rm -v environment-pinned-*.yml
Commit the changes:
git add --all
git commit -m 'add foo package, wipe pinned environments'
git push
Come back here to https://github.com/malariagen/mgenv and make a pull request.
After the PR is created, GitHub actions will automatically run to check that the modified environment can be created, and will generate new versions of the pinned environment files. If CI passes, go to the actions, find and download the artifacts (which will contain new pinned environment files), then add them back in to your clone of the mgenv repo, and push to update the PR.
git add environment-pinned-ubuntu-latest.yml environment-pinned-macos-latest.yml
git commit -m 'add new pinned environment files from CI'
git push
GitHub actions will then run again to verify the pinned environment files can be used to create environments. If CI then passes, the pull request will be merged.