Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update docs #33

Merged
merged 2 commits into from
Jul 25, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,8 @@ docs/*
!docs/conduct_and_info.rst
!docs/whatsnew.rst
!docs/code_structure.rst
!docs/quickstart_legacy_running_compose.rst
!docs/design1.svg
# PyBuilder
target/

Expand Down
17 changes: 17 additions & 0 deletions docs/design1.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
84 changes: 25 additions & 59 deletions docs/quickstart.rst
100644 → 100755
Original file line number Diff line number Diff line change
@@ -1,77 +1,43 @@
Quickstart
==========

1. Make sure you are logged in with a dockerhub account that can access our private dockerhub repository.
To get started with Gingerbread, researchers are required to follow these steps:

``docker login -u gingerbread01``
1. Clone the GitHub repository for the Gingerbread source files by executing the following command in the terminal (Note: Git needs to be installed):

.. important::

You need a password to login, please contact us if you don't have one.

2. Download the source code from github.

``git clone https://github.com/NeoMedSys/gingerbread_sc.git``

3. Make sure you are in the root directory of the project:

``cd gingerbread_sc``

5. Run docker compose to start the container:

with cpu: ``docker compose up -d cpu``

with gpu: ``docker compose up -d gpu``


6. Open a terminal in the container:

``docker exec -it ginger /bin/bash``

.. attention::

Unix systems use a very well defined user system with different permission levels. You might end up with some permission problems if you make new files inside the container and try to access them from your host machine. To avoid this, make sure you always create new files from the host machine and not from inside the container. If the docker container creates the files then you will need to change the permissions of the files to be able to access them from the host machine. To do this, run the following command from the host machine:

Get your user id: ``id -u`` on host machine and ``id -g`` on host machine
```bash
git clone https://github.com/NeoMedSys/gingerbread_sc.git
```

Change the permissions of the files: ``sudo chown -R <user_id>:<group_id> <file_name>``
1. Once the repository is cloned, researchers should make necessary alterations to the *central_processing.py* file, which contains the *prediction_step,* *preprocessing,* and *postprocessing* functions.

for folders: ``sudo chown -R <user_id>:<group_id> <folder_name>``

We are working on a solution to make this process easier.

7. Add medquery environment:

``source /certs/.env``


8. Install the poetry environment:

``poetry install``

.. hint::

If you want to use pytorch-lightning, you will need to add it with ``poetry add pytorch-lightning``. If you want to use CLI version of pytorch-lightning, use ``poetry add pytorch-lightning[extra]``.
You can find the lightning template in /lightning with the corresponding data loader templates.

.. important::
In the *central_processing.py* file, researchers must fill in the required code specific to their model to ensure proper functionality.

.. attention::
The docker image contains a poetry environment:
Add all other code needed in a structured way, use config files and other project structures as baseline of your implementation. Tests will be ran to ensure that the code is working properly with *NeoGate* before uploading.

To run arbitrary code use: ``poetry run python SOMEFILE.py``, this will run the code in the poetry environment. To install new packages, use ``poetry add <package>`` and to remove packages use ``poetry remove <package>``. Read more on poetry here: https://python-poetry.org/docs/basic-usage/.
2. After completing the necessary changes to the *central_processing.py* file use NeoGate to confirm that the code is working properly.

```bash
neogate test -p /gingerbread
```

You are now ready to start coding!
.. important::
The folder needs to be named *gingerbread*, this means basically removing *_sc* from the folder name.

Here is a breakdown of the files in the project:
If the code is working properly, researchers should see the following output:

``central_processing.py``: This is the model template, it uses the basic ``pytorch.nn`` module to define the model. You can work as you would normally do with ``pytorch``, the only difference is that you need to define the ``preprocessing``, ``postprocessing`` and ``predict_step``. We will be using these functions in production, so make sure they are defined correctly and do not change their method names.
```bash
Test passed!
```

``cli_main.py``: This a template for using cli arguments. You can costumize as you want and if you want to use it.
3. The model will the be built appropriately and uploaded to the NeoMedSys Dockerhub repository. To do this, researchers should execute the following command in the terminal:

``data_download.py``: With this you can download data from medquery, check out the documentation here: :doc:`medquery`
.. important::
You need to have docker installed on your machine for this process to work.

``central_processing_handler.py``: This is the bridge between the researchers model and production.
4. Researchers should then create a new GitHub repository for their model and push the code to the new repository.

.. danger::
Do not make any changes to ``central_processing_handler.py``.
.. image:: https://github.com/NeoMedSys/gingerbread_sc/blob/main/docs/design1.svg?raw=true
:align: center
77 changes: 77 additions & 0 deletions docs/quickstart_legacy_running_compose.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
Quickstart
==========

1. Make sure you are logged in with a dockerhub account that can access our private dockerhub repository.

``docker login -u gingerbread01``

.. important::

You need a password to login, please contact us if you don't have one.

2. Download the source code from github.

``git clone https://github.com/NeoMedSys/gingerbread_sc.git``

3. Make sure you are in the root directory of the project:

``cd gingerbread_sc``

5. Run docker compose to start the container:

with cpu: ``docker compose up -d cpu``

with gpu: ``docker compose up -d gpu``


6. Open a terminal in the container:

``docker exec -it ginger /bin/bash``

.. attention::

Unix systems use a very well defined user system with different permission levels. You might end up with some permission problems if you make new files inside the container and try to access them from your host machine. To avoid this, make sure you always create new files from the host machine and not from inside the container. If the docker container creates the files then you will need to change the permissions of the files to be able to access them from the host machine. To do this, run the following command from the host machine:

Get your user id: ``id -u`` on host machine and ``id -g`` on host machine

Change the permissions of the files: ``sudo chown -R <user_id>:<group_id> <file_name>``

for folders: ``sudo chown -R <user_id>:<group_id> <folder_name>``

We are working on a solution to make this process easier.

7. Add medquery environment:

``source /certs/.env``


8. Install the poetry environment:

``poetry install``

.. hint::

If you want to use pytorch-lightning, you will need to add it with ``poetry add pytorch-lightning``. If you want to use CLI version of pytorch-lightning, use ``poetry add pytorch-lightning[extra]``.
You can find the lightning template in /lightning with the corresponding data loader templates.


.. attention::
The docker image contains a poetry environment:

To run arbitrary code use: ``poetry run python SOMEFILE.py``, this will run the code in the poetry environment. To install new packages, use ``poetry add <package>`` and to remove packages use ``poetry remove <package>``. Read more on poetry here: https://python-poetry.org/docs/basic-usage/.


You are now ready to start coding!

Here is a breakdown of the files in the project:

``central_processing.py``: This is the model template, it uses the basic ``pytorch.nn`` module to define the model. You can work as you would normally do with ``pytorch``, the only difference is that you need to define the ``preprocessing``, ``postprocessing`` and ``predict_step``. We will be using these functions in production, so make sure they are defined correctly and do not change their method names.

``cli_main.py``: This a template for using cli arguments. You can costumize as you want and if you want to use it.

``data_download.py``: With this you can download data from medquery, check out the documentation here: :doc:`medquery`

``central_processing_handler.py``: This is the bridge between the researchers model and production.

.. danger::
Do not make any changes to ``central_processing_handler.py``.
4 changes: 4 additions & 0 deletions docs/whatsnew.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
What is new?
============

v0.2.7
------
* Updated Quickstart guide, releasing only inference version of Gingerbread as of now.

v0.2.6
------
* New test structure made in anticipation of NeoGate
Expand Down
2 changes: 1 addition & 1 deletion pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = 'gingerbread'
version = '0.2.6'
version = '0.2.7'
description = 'Researcher template that fits with NeoMedSys'
authors = ['Martin Rovang <martinrovang@gmail.com>']
readme = 'README.md'
Expand Down
Loading