diff --git a/.gitignore b/.gitignore index d3ff1f2..6533556 100644 --- a/.gitignore +++ b/.gitignore @@ -81,6 +81,8 @@ docs/* !docs/conduct_and_info.rst !docs/whatsnew.rst !docs/code_structure.rst +!docs/quickstart_legacy_running_compose.rst +!docs/design1.svg # PyBuilder target/ diff --git a/docs/design1.svg b/docs/design1.svg new file mode 100644 index 0000000..4fce995 --- /dev/null +++ b/docs/design1.svg @@ -0,0 +1,17 @@ + + + + + + + + Source filesBuild files(compose,env,Dockerfiles)GingerbreadNeoGateNoxBuildIf successIf failPublish dockerhubEdited gingerbread source /gingerbreadThe environment variables will be injected directlyinto the docker image, and later we can use a docker-composeto alter some of them, ie. host, ports, etc.Deploy repo that uses the uploaded image as gingerbread source files for deploying model specific apps.Model app running as serviceUse static env variables + additions from a user .yaml file.The static variables can later be controlled from some endpointwhich will also control the versions of the image modules.For now we will keep it at "latest". App versionmodel_owner: name: Martin Soria Røvangmodel_version: v0.1.1.........user_model_info.yamlMRSAL_HOST=http://localhost.comMRSAL_PORT=5000.........system_variablesThese are injects into the Dockerimageas environment variables.ENV MODEL_OWNER=Martin Soria Røvang \ MODEL_VERSION=v0.1.1 \ MRSAL_HOST=http://localhost.com \ MRSAL_PORT=5000 \ ... \ ... \ ... DockerfileNeogate should also upload the model weights and the data used to train the weights. \ No newline at end of file diff --git a/docs/quickstart.rst b/docs/quickstart.rst old mode 100644 new mode 100755 index f40d59a..bbe9504 --- a/docs/quickstart.rst +++ b/docs/quickstart.rst @@ -1,77 +1,43 @@ Quickstart ========== -1. Make sure you are logged in with a dockerhub account that can access our private dockerhub repository. +To get started with Gingerbread, researchers are required to follow these steps: - ``docker login -u gingerbread01`` +1. Clone the GitHub repository for the Gingerbread source files by executing the following command in the terminal (Note: Git needs to be installed): -.. important:: - - You need a password to login, please contact us if you don't have one. - -2. Download the source code from github. - - ``git clone https://github.com/NeoMedSys/gingerbread_sc.git`` - -3. Make sure you are in the root directory of the project: - - ``cd gingerbread_sc`` - -5. Run docker compose to start the container: - - with cpu: ``docker compose up -d cpu`` - - with gpu: ``docker compose up -d gpu`` - - -6. Open a terminal in the container: - - ``docker exec -it ginger /bin/bash`` - -.. attention:: - - Unix systems use a very well defined user system with different permission levels. You might end up with some permission problems if you make new files inside the container and try to access them from your host machine. To avoid this, make sure you always create new files from the host machine and not from inside the container. If the docker container creates the files then you will need to change the permissions of the files to be able to access them from the host machine. To do this, run the following command from the host machine: - Get your user id: ``id -u`` on host machine and ``id -g`` on host machine +```bash +git clone https://github.com/NeoMedSys/gingerbread_sc.git +``` - Change the permissions of the files: ``sudo chown -R : `` +1. Once the repository is cloned, researchers should make necessary alterations to the *central_processing.py* file, which contains the *prediction_step,* *preprocessing,* and *postprocessing* functions. - for folders: ``sudo chown -R : `` - - We are working on a solution to make this process easier. - -7. Add medquery environment: - - ``source /certs/.env`` - - -8. Install the poetry environment: - - ``poetry install`` - -.. hint:: - - If you want to use pytorch-lightning, you will need to add it with ``poetry add pytorch-lightning``. If you want to use CLI version of pytorch-lightning, use ``poetry add pytorch-lightning[extra]``. - You can find the lightning template in /lightning with the corresponding data loader templates. - +.. important:: + In the *central_processing.py* file, researchers must fill in the required code specific to their model to ensure proper functionality. -.. attention:: - The docker image contains a poetry environment: +Add all other code needed in a structured way, use config files and other project structures as baseline of your implementation. Tests will be ran to ensure that the code is working properly with *NeoGate* before uploading. - To run arbitrary code use: ``poetry run python SOMEFILE.py``, this will run the code in the poetry environment. To install new packages, use ``poetry add `` and to remove packages use ``poetry remove ``. Read more on poetry here: https://python-poetry.org/docs/basic-usage/. +2. After completing the necessary changes to the *central_processing.py* file use NeoGate to confirm that the code is working properly. +```bash +neogate test -p /gingerbread +``` -You are now ready to start coding! +.. important:: + The folder needs to be named *gingerbread*, this means basically removing *_sc* from the folder name. -Here is a breakdown of the files in the project: +If the code is working properly, researchers should see the following output: -``central_processing.py``: This is the model template, it uses the basic ``pytorch.nn`` module to define the model. You can work as you would normally do with ``pytorch``, the only difference is that you need to define the ``preprocessing``, ``postprocessing`` and ``predict_step``. We will be using these functions in production, so make sure they are defined correctly and do not change their method names. +```bash +Test passed! +``` -``cli_main.py``: This a template for using cli arguments. You can costumize as you want and if you want to use it. +3. The model will the be built appropriately and uploaded to the NeoMedSys Dockerhub repository. To do this, researchers should execute the following command in the terminal: -``data_download.py``: With this you can download data from medquery, check out the documentation here: :doc:`medquery` +.. important:: + You need to have docker installed on your machine for this process to work. -``central_processing_handler.py``: This is the bridge between the researchers model and production. +4. Researchers should then create a new GitHub repository for their model and push the code to the new repository. -.. danger:: - Do not make any changes to ``central_processing_handler.py``. +.. image:: https://github.com/NeoMedSys/gingerbread_sc/blob/main/docs/design1.svg?raw=true + :align: center \ No newline at end of file diff --git a/docs/quickstart_legacy_running_compose.rst b/docs/quickstart_legacy_running_compose.rst new file mode 100755 index 0000000..f40d59a --- /dev/null +++ b/docs/quickstart_legacy_running_compose.rst @@ -0,0 +1,77 @@ +Quickstart +========== + +1. Make sure you are logged in with a dockerhub account that can access our private dockerhub repository. + + ``docker login -u gingerbread01`` + +.. important:: + + You need a password to login, please contact us if you don't have one. + +2. Download the source code from github. + + ``git clone https://github.com/NeoMedSys/gingerbread_sc.git`` + +3. Make sure you are in the root directory of the project: + + ``cd gingerbread_sc`` + +5. Run docker compose to start the container: + + with cpu: ``docker compose up -d cpu`` + + with gpu: ``docker compose up -d gpu`` + + +6. Open a terminal in the container: + + ``docker exec -it ginger /bin/bash`` + +.. attention:: + + Unix systems use a very well defined user system with different permission levels. You might end up with some permission problems if you make new files inside the container and try to access them from your host machine. To avoid this, make sure you always create new files from the host machine and not from inside the container. If the docker container creates the files then you will need to change the permissions of the files to be able to access them from the host machine. To do this, run the following command from the host machine: + + Get your user id: ``id -u`` on host machine and ``id -g`` on host machine + + Change the permissions of the files: ``sudo chown -R : `` + + for folders: ``sudo chown -R : `` + + We are working on a solution to make this process easier. + +7. Add medquery environment: + + ``source /certs/.env`` + + +8. Install the poetry environment: + + ``poetry install`` + +.. hint:: + + If you want to use pytorch-lightning, you will need to add it with ``poetry add pytorch-lightning``. If you want to use CLI version of pytorch-lightning, use ``poetry add pytorch-lightning[extra]``. + You can find the lightning template in /lightning with the corresponding data loader templates. + + +.. attention:: + The docker image contains a poetry environment: + + To run arbitrary code use: ``poetry run python SOMEFILE.py``, this will run the code in the poetry environment. To install new packages, use ``poetry add `` and to remove packages use ``poetry remove ``. Read more on poetry here: https://python-poetry.org/docs/basic-usage/. + + +You are now ready to start coding! + +Here is a breakdown of the files in the project: + +``central_processing.py``: This is the model template, it uses the basic ``pytorch.nn`` module to define the model. You can work as you would normally do with ``pytorch``, the only difference is that you need to define the ``preprocessing``, ``postprocessing`` and ``predict_step``. We will be using these functions in production, so make sure they are defined correctly and do not change their method names. + +``cli_main.py``: This a template for using cli arguments. You can costumize as you want and if you want to use it. + +``data_download.py``: With this you can download data from medquery, check out the documentation here: :doc:`medquery` + +``central_processing_handler.py``: This is the bridge between the researchers model and production. + +.. danger:: + Do not make any changes to ``central_processing_handler.py``. diff --git a/docs/whatsnew.rst b/docs/whatsnew.rst index f74f2d7..58c0a27 100644 --- a/docs/whatsnew.rst +++ b/docs/whatsnew.rst @@ -1,6 +1,10 @@ What is new? ============ +v0.2.7 +------ +* Updated Quickstart guide, releasing only inference version of Gingerbread as of now. + v0.2.6 ------ * New test structure made in anticipation of NeoGate diff --git a/pyproject.toml b/pyproject.toml index 0c3c491..1064c0e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = 'gingerbread' -version = '0.2.6' +version = '0.2.7' description = 'Researcher template that fits with NeoMedSys' authors = ['Martin Rovang '] readme = 'README.md'