Documentation module (#193)

* documentation

* include reference

* fix doc linking

* fix doc linking

* fix doc linking

.pre-commit-config.yaml

@@ -8,7 +8,7 @@ repos:
       - id: end-of-file-fixer
       - id: trailing-whitespace
       # - id: check-json
-      - id: check-yaml
+      # - id: check-yaml
       - id: check-toml
       - id: check-docstring-first
       - id: check-executables-have-shebangs

mkdocs.yml

@@ -28,6 +28,7 @@ theme:
     - navigation.instant
     - navigation.indexes
     - content.code.copy
+    - content.code.annotate
     - navigation.footer
     - search.suggest
     - search.highlight
@@ -50,6 +51,9 @@ markdown_extensions:
   - pymdownx.tasklist:
       custom_checkbox: true
   - pymdownx.superfences
+  - pymdownx.emoji:
+      emoji_index: !!python/name:material.extensions.emoji.twemoji
+      emoji_generator: !!python/name:material.extensions.emoji.to_svg
 
 plugins:
   - search
@@ -120,8 +124,9 @@ nav:
   - S10 - Extra 🔥:
     - s10_extra/README.md
     - M30 - Command Line Interfaces: s10_extra/cli.md
-    - M31 - Hyperparameter optimization: s10_extra/hyperparameters.md
-    - M32 - High Performance Clusters: s10_extra/high_performance_clusters.md
+    - M31 - Documentation: s10_extra/documentation.md
+    - M32 - Hyperparameter optimization: s10_extra/hyperparameters.md
+    - M33 - High Performance Clusters: s10_extra/high_performance_clusters.md
   - Summary: overview.md
   - Projects: projects.md
   - Challenges: challenges.md

s10_extra/README.md

@@ -2,3 +2,10 @@
 
 All modules listed here are not part of the core course, but expands on some of the other topics.
 Some of them may still be under construction and may in the future be moved into other sessions.
+
+<p align="center">
+  <img src="../figures/icons/click.png" width="130">
+  <img src="../figures/icons/material.png" width="130">
+  <img src="../figures/icons/optuna.png" width="130">
+  <img src="../figures/icons/pbs.png" width="130">
+</p>

s10_extra/documentation.md

@@ -0,0 +1,318 @@
+![Logo](../figures/icons/material.png){ align=right width="130"}
+
+# Documentation
+
+---
+
+In today's rapidly evolving software development landscape, effective documentation is a crucial component of any
+project. The ability to create clear, concise, and user-friendly technical documentation can make a significant
+difference in the success of your codebase. We all probably encountered code that we wanted to use, only for us to
+abandon using it because it was missing documentation such that we could get started with it.
+
+Technical documentation or code documentation can be many things:
+
+* Plain text, images and videos explaining core concepts for your software
+* Documentation of API on how to call a function or class, what the different parameters are etc.
+* Code examples of how to use certain functionality
+
+and many more. We are in this module going to focus on setting up a very basic documentation system that will
+automatically help you document the API of your code. For this reason we recommend that before continuning with this
+module that you have completed
+[module M7 on good coding practices](../s2_organisation_and_version_control/good_coding_practice.md)
+or have similar experience with writing docstrings for python functions and classes.
+
+There are different systems for writing documentation. In fact there is a lot to choose from:
+
+* [MkDocs](https://www.mkdocs.org/)
+* [Sphinx](https://www.sphinx-doc.org/en/master/)
+* [GitBook](https://www.gitbook.com/)
+* [Docusaurus](https://docusaurus.io/)
+* [Doxygen](https://www.doxygen.nl/)
+* [Jekyll](https://jekyllrb.com/)
+
+Important to note that all these are *static site generators*. The word *static* here refers to that when the content
+is generated and served on a webside, the underlying HTML code will not change. It may contain HTML elements that
+dynamic (like video), but the site does not change (1).
+{ .annotate }
+
+1. :man_raising_hand: Good examples of dynamic sites are any social media or news media where new posts, pages etc.
+    are constantly added over time. Good examples of static sites are documentation, blogposts etc.
+
+We are in this module going to look at [Mkdocs](https://www.mkdocs.org/), which (in my opinion) is one of the easiest
+systems to get started with because all documentation is written in markdown and the build system is written in Python.
+As an alternativ, you can consider doing the exercises in [Sphinx](https://www.sphinx-doc.org/en/master/) which is
+probably the most used documentation system for Python code. Sphinx offer more customization than Mkdocs, so is
+generally preferred for larger projects with complex documentation, but for smaller projects Mkdocs should be easier to
+get started with and is sufficient.
+
+Mkdocs by default does not include many features and for that reason we are directly going to dive into using the
+[material for mkdocs theme](https://github.com/squidfunk/mkdocs-material) that provides a lot of nice customization
+to create professional static sites. In fact, this hole course is written in mkdocs using the material theme.
+
+## Mkdocs
+
+The core file when using mkdocs is the `mkdocs.yml` file, which is the configuration file for the project:
+
+```yaml
+site_name: Documentation of my project
+site_author: Jane Doe
+docs_dir: source # (1)!
+
+theme:
+    language: en
+    name: material # (2)!
+    features: # (3)!
+    - content.code.copy
+    - content.code.annotate
+
+plugins: # (4)!
+    - search
+    - mkdocstrings
+
+nav: # (5)!
+  - Home: index.md
+
+```
+
+1. :man_raising_hand: This indicates the source directory of our documentation. If the layout of your documentation is
+    a bit different than what described above, you may need to change this.
+
+2. :man_raising_hand: The overall theme of your documentation. We recommend the `material` theme but there are
+    [many more to choose from](https://github.com/mkdocs/mkdocs/wiki/MkDocs-Themes) and you can also
+    [create your own](https://www.mkdocs.org/dev-guide/themes/).
+
+3. :man_raising_hand: The `featuers` section is where features that are supported by your given theme can be enabled.
+    In this example we have enabled `content.code.copy` feature which adds a small copy button to all code block and the
+    `content.code.annotate` feature which allows you to add annotations like this box to code blocks.
+
+4. :man_raising_hand: [Plugins](https://www.mkdocs.org/dev-guide/plugins/) add new functionality to your documentation.
+    In this case we have added two plugins that add functionality for searching through our documentation and
+    automatically adding documentation from docstrings. Remember that some plugins requires you to install additional
+    Python packages with those plugins, so remember to add them to your `requirements.txt` file.
+
+5. :man_raising_hand: The `nav` section is where you define the navigation structure of your documentation. When you
+    add new `.md` files to the `source` folder you then need to add them to the `nav` section.
+
+And that is more or less what you need to get started. In general, if you need help with configuration of your
+documentation in mkdocs I recommend looking at [this page](https://www.mkdocs.org/getting-started/) and
+[this page](https://squidfunk.github.io/mkdocs-material/getting-started/).
+
+## Exercises
+
+In this set of exercises we assume that you have completed
+[module M6 on code structure](../s2_organisation_and_version_control/code_structure.md) and therefore have a repository
+that at least contains the following:
+
+```txt
+├── pyproject.toml     <- Project configuration file with package metadata
+│
+├── docs               <- Documentation folder
+│   │
+│   ├── index.md       <- Homepage for your documentation
+│   │
+│   ├── mkdocs.yml     <- Configuration file for mkdocs
+│   │
+│   └── source/        <- Source directory for documentation files
+│
+└── src                <- Source code for use in this project.
+│   │
+│   ├── __init__.py    <- Makes src a Python module
+│   │
+│   ├── models         <- model implementations, training script
+│   │   ├── __init__.py
+│   │   ├── model.py
+│   │   ├── train_model.py
+...
+```
+
+It is not important exactly what is in the `src` folder for the exercises, but we are going to refer to the above
+structure in the exercises, so adjust accordingly if you diviate from this. Additionally, we are going to assume that
+your project code is installed in your environment such that it can be imported as normal python code.
+
+1. We are going to need two python packages to get started: [mkdocs](https://pypi.org/project/mkdocs/) and
+    [material for mkdocs](https://pypi.org/project/mkdocs-material/). Install with
+
+    ```bash
+    pip install "mkdocs-material >= 4.8.0" # (1)!
+    ```
+
+    1. Since `mkdocs` is a dependency of `mkdocs-material` we only need to install the latter.
+
+2. Run in your terminal (from the `docs` folder):
+
+    ```bash
+    mkdocs serve # (1)!
+    ```
+
+    1. :man_raising_hand: `mkdocs serve` will automatically rebuild the hole site whenever you save a file inside the
+        `docs` folder. This is not a problem if you have a fairly small site with not that many pages (or elements), but
+        can take a long time for large sites. Consider running with the `--dirty` option for only re-building the site
+        for files that have been changed.
+
+    which should render the `index.md` file as the homepage. You can leave the documentation server running during the
+    remaining exercises.
+
+3. We are no ready to document the API of our code:
+
+    1. Make sure you at least have one function and class inside your `src` module. If you do not have you can for
+        simplicity copy the following module to the `src/models/model.py` file
+
+        ```python
+        import torch
+
+        class MyNeuralNet(torch.nn.Module):
+            """Basic neural network class.
+
+            Args:
+                in_features: number of input features
+                out_features: number of output features
+
+            """
+            def __init__(self, in_features: int, out_features: int) -> None:
+                self.l1 = torch.nn.Linear(in_features, 500)
+                self.l2 = torch.nn.Linear(500, out_features)
+                self.r = torch.nn.ReLU()
+
+            def forward(self, x: torch.Tensor) -> torch.Tensor:
+                """Forward pass of the model.
+
+                Args:
+                    x: input tensor expected to be of shape [N,in_features]
+
+                Returns:
+                    Output tensor with shape [N,out_features]
+
+                """
+                return self.l2(self.r(self.l1(x)))
+
+        ```
+
+        and the following function to add `src/predict_model.py` file:
+
+        ```python
+
+        def predict(
+            model: torch.nn.Module,
+            dataloader: torch.utils.data.DataLoader
+        ) -> None:
+            """Run prediction for a given model and dataloader.
+
+            Args:
+                model: model to use for prediction
+                dataloader: dataloader with batches
+
+            Returns
+                Tensor of shape [N, d] where N is the number of samples and d is the output dimension of the model
+
+            """
+            return [model(batch) for batch in dataloader]
+        ```
+
+    2. Add a markdown file to the `docs/source` folder called `my_api.md` and add that file to the `nav:` section in
+        the `mkdocs.yaml` file.
+
+    3. To that file add the following code:
+
+        ```markdown
+
+        # My API
+
+        ::: src.models.model.MyNeuralNet
+
+        ::: src.predict_model.predict
+
+        ```
+
+        The `:::` indicator tells mkdocs that it should look for the corresponding function/module and then render it
+        on the given page. Thus, if you have a function/module located in another location change the paths accordingly.
+
+    4. Make sure that the documentation correctly includes your function and module on the given page.
+
+    5. (Optional) Include more functions/modules in your documentation.
+
+4. (Optional) Look through the documentation for [mkdocstrings](https://mkdocstrings.github.io/python/) and try to
+    improve the layout a bit. Especially, the
+    [headings](https://mkdocstrings.github.io/python/usage/configuration/headings/),
+    [docstrings](https://mkdocstrings.github.io/python/usage/configuration/docstrings/) and
+    [signatures](https://mkdocstrings.github.io/python/usage/configuration/signatures/) could be of interest to adjust.
+
+5. Finally, try to build a final version of your documentation
+
+    ```bash
+    mkdocs build
+    ```
+
+    this should result in a `site` folder that contains the actual HTML code for documentation.
+
+## Publish your documentation
+
+To publish your documentation you need a place to host your build documentation e.g. the content of the `site` folder
+you build in the last exercise. There are many places to host your documentation, but if you only need a static site
+and are already hosting your code through Github, then a good option is [Github Pages](https://pages.github.com/).
+Github pages is free to use for your public projects.
+
+Before getting started with this set of exercises you should have completed
+[module M16 on github actions](../s5_continuous_integration/github_actions.md) so you already know about workflow files.
+
+### Exercises
+
+1. Start by adding a new file called `deploy_docs.yaml` to the `.github/workflows` folder. Add the following cod to that
+    file and save it.
+
+    ```yaml
+    name: Deploy docs
+
+    on:
+    push:
+        branches:
+            - main
+
+    permissions:
+        contents: write # (1)
+
+    jobs:
+    deploy:
+        runs-on: ubuntu-latest
+        steps:
+            - uses: actions/checkout@v3
+              with:
+                fetch-depth: 0
+            - uses: actions/setup-python@v4
+              with:
+                python-version: 3.10
+            - uses: actions/cache@v2
+              with:
+                key: ${{ github.ref }}
+                path: .cache
+            - run: pip install -r requirements.txt
+            - run: mkdocs gh-deploy --force
+    ```
+
+    1. :man_raising_hand: It is important to give `write` premissions to this actions because it is not only reading
+        your code but it will actually also push code.
+
+    Before continuing, make sure you understand what the different steps of the workflow does and especially we
+    recommend looking at the documentation of the `mkdocs gh-deploy` command.
+
+2. Commit and push the file. Check that the action is executed and if it succeeds, that your build project is pushed to
+    a branch called `gh-pages`. If the action does not succeeds, then figure out what is wrong and fix it!
+
+3. After confirming that our action is working, you need to configure Github to actually publish the content being
+    build by Github Actions. Do the following:
+
+    * Go to the Settings tab and then the Pages subsection
+    * In the `Source` setting choose the `Deploy from a branch`
+    * In the `Branch` setting choose the `gh-pages` branch and `/(root)` folder and save
+
+    <figure markdown>
+        ![Image](../figures/github_pages.png){ width="700" }
+    </figure>
+
+    This should then start deploying your site to `https://<your-username>.github.io/<your-reponame>/`. If it does not
+    do this you may need to recommit and trigger the github actions build again.
+
+4. Make sure your documentation is published and looks as it should.
+
+This ends the module on technical documentation. We cannot stress enough how important it is to write proper
+documentation for larger projects that need to be maintained over a longer time. It is often a iterative process, but
+it is often best to do it while writing the code.

s5_continuous_integration/github_actions.md

@@ -229,8 +229,12 @@ we will setup github workflows that will automatically test for this.
     to get passing, so this exercise formally only requires you to get `isort` and `flake8`
     passing.
 
-This ends the module on Github workflows. If you have not already stumbled across this feature, if you try to create
-a workflow file directly in Github you may encounter the following page
+This ends the module on Github workflows. If you are more interested in this topic you can checkout module
+[M31 on documentation](../s10_extra/documentation.md) which first including locally building some documentation for your
+project and afterwards use Github actions for deploying it to Github Pages.
+
+If you have not already stumbled across this feature, if you try to create a workflow file directly in Github you may
+encounter the following page
 
 ![action](../figures/github_workflows.PNG)