Merge pull request #83 from HelmholtzAI-Consultants-Munich/models-ref…

…actoring

Models refactoring

.github/workflows/test.yml

@@ -44,17 +44,15 @@ jobs:
         run: |
           python -m pip install --upgrade pip
           python -m pip install setuptools  
-          pip install pytest
-          pip install pytest-qt
           pip install pytest-xvfb
           pip install coverage
-          pip install -e ".[testing]"
+          pip install -e ".[dev]"
           pip install matplotlib
         working-directory: src/client
 
       - name: Install server dependencies (for communication tests)
         run: |
-          pip install -e ".[testing]"
+          pip install -e ".[dev]"
         working-directory: src/server
 
       - name: Test with pytest
@@ -94,10 +92,9 @@ jobs:
           python -m pip install --upgrade pip
           python -m pip install --upgrade setuptools  
           pip install numpy
-          pip install pytest
           pip install wheel
           pip install coverage
-          pip install -e ".[testing]"
+          pip install -e ".[dev]"
         working-directory: src/server
 
       - name: Test with pytest
@@ -115,7 +112,6 @@ jobs:
         env:
           CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
 
-
   deploy:
       # this will run when you have tagged a commit, starting with "v*"
       # and requires that you have put your twine API key in your 

.gitignore

@@ -21,7 +21,7 @@ __pycache__/
 
 # Distribution / packaging
 .Python
-# build/
+build/
 develop-eggs/
 dist/
 downloads/
@@ -153,4 +153,6 @@ dmypy.json
 .idea/
 
 .DS_Store
+data/
+BentoML/
 

README.md

@@ -29,5 +29,4 @@ Our platform encourages the use of data centric practices. With the user friendl
 - Focus on data curation: no interaction with model parameters during training and inference
 
 #### *Get more with less!*
-
 <img src="https://github.com/HelmholtzAI-Consultants-Munich/data-centric-platform/blob/main/src/client/readme_figs/dcp_pipeline.png"  width="200" height="200">

docs/source/conf.py

@@ -25,7 +25,7 @@
 # https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
 
 html_theme = 'alabaster'
-html_static_path = ['_static']
+#html_static_path = ['_static']
 
 import os
 import sys
@@ -35,7 +35,7 @@
 # Add parent dir to known paths
 p = Path(__file__).parents[2]
 sys.path.insert(0, os.path.abspath(p))
-
+sys.path.insert(0, os.path.join(p, 'src/server/dcp_server'))
 # Add the following extensions
 extensions = [
     'sphinx.ext.autodoc',

docs/source/dcp_client.rst

@@ -1,13 +1,32 @@
 dcp\_client package
 ===================
 
+The dcp_client package contains modules and subpackages for interacting with a server for model inference and training. It provides functionalities for managing GUI windows, handling image storage, and connecting to the server for model operations.
 
-.. toctree::
-   :maxdepth: 4
+dcp_client.app
+   Defines the core application class and related functionalities.
+    - ``dcp_client.app.Application``: Represents the main application and provides methods for image management, model interaction, and server connectivity.
+    - ``dcp_client.app.DataSync``: Abstract base class for data synchronization operations.
+    - ``dcp_client.app.ImageStorage``: Abstract base class for image storage operations.
+    - ``dcp_client.app.Model``: Abstract base class for model operations.
 
-   dcp_client.gui
-   dcp_client.utils
+dcp_client.gui
+   Contains modules for GUI components.
+    - ``dcp_client.gui.main_window``: Defines the main application window and associated event functions.
+    - ``dcp_client.gui.napari_window``: Manages the Napari window and its functionalities.
+    - ``dcp_client.gui.welcome_window``: Implements the welcome window and its interactions.
 
+dcp_client.utils
+   Contains utility modules for various tasks.
+    - ``dcp_client.utils.bentoml_model``: Handles interactions with BentoML for model inference and training.
+    - ``dcp_client.utils.fsimagestorage``: Provides functions for managing images stored in the filesystem.
+    - ``dcp_client.utils.settings``: Defines initialization functions and settings.
+    - ``dcp_client.utils.sync_src_dst``: Implements data synchronization between source and destination.
+    - ``dcp_client.utils.utils``: Offers various utility functions for common tasks.
+
+
+Submodules
+----------
 
 dcp\_client.app module
 ----------------------
@@ -17,3 +36,17 @@ dcp\_client.app module
    :undoc-members:
    :show-inheritance:
 
+dcp\_client.gui module
+----------------------
+.. toctree::
+   :maxdepth: 4
+
+   dcp_client.gui
+
+dcp\_client.utils module
+------------------------
+.. toctree::
+   :maxdepth: 4
+
+   dcp_client.utils
+

docs/source/dcp_client_installation.rst

@@ -114,7 +114,7 @@ DCP Shortcuts
 - In the Data Overview window, clicking on an image and the hitting the **Enter** key, is equivalent to clicking the 'View Image and Fix Label' button
 - The viewer accepts all Napari Shortcuts. The current list of the shortcuts for macOS can be see below:
 
-.. image:: https://raw.githubusercontent.com/HelmholtzAI-Consultants-Munich/data-centric-platform/add-documentation/src/client/readme_figs/napari_shortcuts.png
+.. image:: https://raw.githubusercontent.com/HelmholtzAI-Consultants-Munich/data-centric-platform/main/src/client/readme_figs/napari_shortcuts.png
    :width: 600
    :height: 500
    :align: center

docs/source/dcp_server.rst

@@ -1,23 +1,25 @@
 dcp\_server package
 ===================
 
-.. automodule:: dcp_server
-   :members:
-   :undoc-members:
-   :show-inheritance:
-   :exclude-members: dcp\_server.main module
+The dcp_server package is structured to handle various server-side functionalities related model serving for segmentation and training. 
 
-Submodules
-----------
+dcp_server.models
+    Defines various models for cell classification and segmentation, including CellClassifierFCNN, CellClassifierShallowModel, CellposePatchCNN, CustomCellposeModel, and UNet.
+    These models handle tasks such as evaluation, forward pass, training, and updating configurations.
 
-dcp\_server.fsimagestorage module
----------------------------------
+dcp_server.segmentationclasses
+    Defines segmentation classes for specific projects, such as GFPProjectSegmentation, GeneralSegmentation, and MitoProjectSegmentation.
+    These classes contain methods for segmenting images and training models on images and masks.
 
-.. automodule:: dcp_server.fsimagestorage
-   :members:
-   :undoc-members:
-   :show-inheritance:
+dcp_server.serviceclasses
+    Defines service classes, such as CustomBentoService and CustomRunnable, for serving the models with BentoML and handling computation on remote Python workers.
 
+dcp_server.utils
+    Provides various utility functions for dealing with image storage, image processing, feature extraction, file handling, configuration reading, and path manipulation.
+
+
+Submodules
+----------
 
 dcp\_server.models module
 -------------------------
@@ -44,9 +46,9 @@ dcp\_server.serviceclasses module
    :show-inheritance:
 
 dcp\_server.utils module
-------------------------
+---------------------------------
 
-.. automodule:: dcp_server.utils
-   :members:
-   :undoc-members:
-   :show-inheritance:
+.. toctree::
+   :maxdepth: 4
+
+   dcp_server.utils

docs/source/dcp_server.utils.rst

@@ -0,0 +1,37 @@
+dcp\_server.utils package
+=========================
+
+.. automodule:: dcp_server.utils
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Submodules
+----------
+
+dcp\_server.utils.fsimagestorage module
+---------------------------------------
+
+.. automodule:: dcp_server.utils.fsimagestorage
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+dcp\_server.utils.helpers module
+---------------------------------------
+
+.. automodule:: dcp_server.utils.helpers
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+dcp\_server.utils.processing module
+-----------------------------------
+
+.. automodule:: dcp_server.utils.processing
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+
+

docs/source/dcp_server_installation.rst

@@ -19,7 +19,7 @@ Before starting make sure you have navigated to ``data-centric-platform/src/serv
 
 .. code-block:: bash
 
-   pip install -e .
+   pip install -e ".[dev]"
 
 Launch DCP Server
 ------------------
@@ -83,17 +83,18 @@ The models are currently integrated into DCP:
 
 - **Instance** Segmentation: 
 
-  - ``CustomCellposeModel``: Inherits from cellpose.models.CellposeModel, see `here <https://cellpose.readthedocs.io/en/latest/api.html#cellposemodel>`__ for more information.
+  - ``CustomCellpose``: Inherits from cellpose.models.CellposeModel, see `here <https://cellpose.readthedocs.io/en/latest/api.html#cellposemodel>`__ for more information.
 - **Semantic** Segmentation: 
 
   - ``UNet``: A vanilla U-Net model, trained on the full images
-- **Panoptic** Segmentation: 
+- **Multi Class Instance** Segmentation: 
 
-  - ``CellposePatchCNN``: Includes a segmentor for instance segmentation, sequentially followed by a classifier for semantic segmentation. The segmentor can only be ``CustomCellposeModel`` model, while the classifier can be one of:
+  - ``Inst2MultiSeg``: Includes a segmentor for instance segmentation, sequentially followed by a classifier for semantic segmentation. The segmentor can only be ``CustomCellposeModel`` model, while the classifier can be one of:
 
-    - ``CellClassifierFCNN`` or "FCNN" (in config): A CNN model for obtaining class labels, trained on images patches of individual objects, extarcted using the instance mask from the previous step
-    - ``CellClassifierShallowModel`` or "RandomForest" (in config): A Random Forest model for obtaining class labels, trained on shape and intensity features of the objects, extracted using the instance mask from the previous step.
-  - UNet: If the post-processing argument is set, then the instance mask is deduced from the labels mask. Will not be able to handle touching objects 
+    - ``PatchClassifier`` or "FCNN" (in config): A CNN model for obtaining class labels, trained on images patches of individual objects, extarcted using the instance mask from the previous step
+    - ``FeatureClassifier`` or "RandomForest" (in config): A Random Forest model for obtaining class labels, trained on shape and intensity features of the objects, extracted using the instance mask from the previous step.
+  - ``MultiCellpose``: Includes **n** CustomCellpose models, where n equals the number of classes, stacked such that each model predicts only the object corresponding to each class.
+  - ``UNet``: If the post-processing argument is set, then the instance mask is deduced from the labels mask. Will not be able to handle touching objects 
 
 
 Running with Docker 

docs/source/index.rst

@@ -31,7 +31,7 @@ DCP handles all kinds of **segmentation tasks**! Try it out if you need to do:
 
 - **Instance** segmentation
 - **Semantic** segmentation
-- **Panoptic** segmentation
+- **Multi-class instance** segmentation
 
 Toy data
 --------

src/client/MANIFEST.in

@@ -1 +1 @@
-include dcp_client/*.cfg
+include dcp_client/*.yaml

src/client/dcp_client/__init__.py

@@ -39,4 +39,4 @@
 
 This package structure allows for easy management of GUI components, image storage, model interactions, and server connectivity within the dcp_client application.
 
-"""
+"""