Merge branch 'helmholtz_design' of https://github.com/HelmholtzAI-Con…

…sultants-Munich/data-centric-platform into helmholtz_design

docs/source/dcp_client_installation.rst

@@ -11,12 +11,28 @@ The client of our data centric platform for microscopy imaging.
 Installation
 -------------
 
-Before starting make sure you have navigated to ``data-centric-platform/src/client``. All future steps expect you are in the client directory. This installation has been tested using a conda environment with python version 3.9 on a mac local machine. In your dedicated environment run:
+For installing dcp-client you will first need to clone the repo:
+
+.. code-block:: bash
+
+   git clone https://github.com/HelmholtzAI-Consultants-Munich/data-centric-platform.git
+
+
+Then navigate to the client directory:
+
+.. code-block:: bash
+
+   cd data-centric-platform/src/client
+
+In your dedicated environment run:
 
 .. code-block:: bash
 
    pip install -e .
 
+
+This installation has been thoroughly tested using a conda environment with python version 3.9, 3.10, 3.11 and 3.12 on a macOS local machine.
+
 Running the client: A step-by-step guide!
 ------------------------------------------
 
@@ -25,13 +41,20 @@ Running the client: A step-by-step guide!
 
 DCP includes a client and server side for using our data centric platform. The client and server communicate via the `bentoml <https://www.bentoml.com/?gclid=Cj0KCQiApKagBhC1ARIsAFc7Mc6iqOLi2OcLtqMbGx1KrFjtLUEZ-bhnqlT2zWREE0x7JImhtNmKlFEaAvSSEALw_wcB>`_ library. 
    There are currently two options available: running the server locally, or connecting to the running instance on the FZJ jusuf-cloud.
-   Before continuing, you need to make sure that DCP server is running, either locally or on the cloud. See :doc: `dcp_server_installation` for instructions on how to launch the server. **Note:** In order for this connection to succeed, you will need to have contacted the team developing DCP, so they can add your IP to the list of accepted requests.
+   Before continuing, you need to make sure that DCP server is running, either locally or on the cloud. See :ref:`DCP Client` for instructions on how to launch the server. **Note:** In order for this connection to succeed, you will need to have contacted the team developing DCP, so they can add your IP to the list of accepted requests.
 
 After you are certain the server is running, simply run:
 
    .. code-block:: bash
 
-      dcp-client --mode local/remote
+      dcp-client --mode local
+
+   or
+
+   .. code-block:: bash
+
+      dcp-client --mode remote
+
    
 Set the ``--mode`` argument to ``local`` or ``remote`` depending on which setup you have chosen for the server.
 
@@ -60,12 +83,9 @@ Set the ``--mode`` argument to ``local`` or ``remote`` depending on which setup
 
    This folder is intended to contain images along with their final segmentations. **Only** move images here when the segmentation is complete and finalised, you won't be able to change them after they have been moved here. These are then used for training your model.
 
-3. **Setting paths**
-~~~~~~~~~~~~~~~~~~~~~
-
    After setting the paths for these three folders, you can click the **Start** button. If you have set the server configuration to the cloud, you will receive a message notifying you that your data will be uploaded to the cloud. Click **Ok** to continue.
 
-4. **Data Overview**
+3. **Data Overview**
 ~~~~~~~~~~~~~~~~~~~~
 
    The main working window will appear next. This gives you an overview of the directories selected in the previous step along with three options:
@@ -79,10 +99,10 @@ Set the ``--mode`` argument to ``local`` or ``remote`` depending on which setup
       :height: 200
       :align: center
 
-5. **The viewer**
+4. **The viewer**
 ~~~~~~~~~~~~~~~~~~~~
 
-   In DCP, we use [napari](https://napari.org/stable) for viewing our images and masks, adding, editing or removing labels. An example of the viewer can be seen below. After adding or removing any objects and editing existing objects wherever necessary, there are two options available:
+   In DCP, we use  `napari <https://napari.org/stable>`_ for viewing our images and masks, adding, editing or removing labels. The newest version of DCP comes with `napari-sam <https://www.napari-hub.org/plugins/napari-sam>`_ integration, to enable even faster labelling! Check out the turorial on the napari-hub to find out how to use this tool. An example of the viewer can be seen below. After adding or removing any objects and editing existing objects wherever necessary, there are two options available:
 
    - Click the **Move to Curation in progress folder** if you are not 100% certain about the labels you have created. You can also click on the label in the labels layer and change the name. This will result in several label files being created in the *In progress folder*, which can be examined later on.  **Note:** When changing the layer name in Napari, the user should rename it such that they add their initials or any other new info after _seg. E.g., if the labels of 1_seg.tiff have been changed in the Napari viewer, then the appropriate naming would for example be: 1_seg_CB.tiff and not 1_CB_seg.tiff.
    - Click the **Move to Curated dataset folder** if you are certain that the labels you are now viewing are final and require no more curation. These images and labels will later be used for training the machine learning model, so make sure that you select this option only if you are certain about the labels. If several labels are displayed (opened from the 'Curation in progress' step), make sure to **click** on the single label in the labels layer list you wish to be moved to the *Curated data folder*. The other images will then be automatically deleted from this folder.

docs/source/dcp_server_installation.rst

@@ -15,11 +15,28 @@ The client and server communicate via the `bentoml <https://www.bentoml.com/?gcl
 Installation
 --------------
 
-Before starting make sure you have navigated to ``data-centric-platform/src/server``. All future steps expect you are in the server directory. In your dedicated environment run:
+For installing dcp-server you will need to have Python <3.12, the repo has been tested on Python versions 3.9, 3.10 and 3.11 on latest versions of Windows, Ubuntu and MacOS. 
+
+First clone the repo:
+
+.. code-block:: bash
+
+   git clone https://github.com/HelmholtzAI-Consultants-Munich/data-centric-platform.git
+
+Then navigate to the server directory:
 
 .. code-block:: bash
 
-   pip install -e ".[dev]"
+   cd data-centric-platform/src/server
+
+
+In your dedicated environment run:
+
+.. code-block:: bash
+
+   pip install numpy
+   pip install -e .
+
 
 Launch DCP Server
 ------------------

scripts/openstack/02_setup_vm.sh

@@ -1,59 +1,24 @@
 # Connect to the machine via ssh
-# ssh -v -i ~/.ssh/id_rsa_project1 ubuntu@134.94.88.118
-# ssh -v -i ~/.ssh/id_rsa_project2 ubuntu@134.94.88.74
+# ssh -v -i ~/.ssh/id_rsa rocky@134.94.198.230 # IP should reflect the created floatingIP
 
-# Updates
-echo "Updating distro"
-sudo apt update && sudo apt upgrade
+# Bring VM to latest state and install some dependencies
+sudo dnf update -y # cosmetic for getting the latest kernel in
+sudo shutdown 1 -r
+sudo dnf config-manager --set-enabled crb
+sudo dnf -y install epel-release
+sudo dnf install -y gcc gcc-c++ make kernel-headers-$(uname -r) kernel-devel-$(uname -r) tar bzip2 automake elfutils-libelf-devel libglvnd libglvnd-devel libglvnd-opengl libglvnd-glx acpid pciutils dkms
 
-# Install NVIDIA drivers
-echo "Installing NVIDIA driver"
-sudo apt install -y gcc make g++
-sudo apt-get install linux-headers-$(uname -r)
-curl -o /tmp/NVIDIA-Driver.latest.run https://hpsrepo.fz-juelich.de/jusuf/nvidia/NVIDIA-Driver.latest
-chmod 755 /tmp/NVIDIA-Driver.latest.run
-sudo mkdir /etc/nvidia
-curl -o /tmp/gridd.conf https://hpsrepo.fz-juelich.de/jusuf/nvidia/gridd.conf
-sudo mv /tmp/gridd.conf /etc/nvidia/gridd.conf
-sudo /tmp/NVIDIA-Driver.latest.run --ui=none --no-questions --disable-nouveau
-# sudo reboot # run if needed
+# Install Nvidia driver
+sudo dnf config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel9/$(uname -i)/cuda-rhel9.repo
+sudo dnf module -y install nvidia-driver:latest-dkms
+sudo shutdown 1 -r
 
-# Add the following to ~/.bashrc
-#export PATH=/usr/local/cuda-11.8/bin${PATH:+:${PATH}}
-#export LD_LIBRARY_PATH=/usr/local/cuda-11.8/lib\
-#                         ${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
-# alias labelapp="cd /home/ubuntu/active-learning-platform && python al_framework.py"
+# After rebooting the device, check the driver is instlled correctly
+nvidia-smi
 
 # Install miniconda
-echo "Installing Miniconda"
-wget -O /tmp/Miniconda3-latest-Linux-x86_64.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
-bash /tmp/Miniconda3-latest-Linux-x86_64.sh
-
-# Install napari
-# conda env remove -n .venv
-echo "Installing napari"
-conda activate base && pip install napari[all]
-sudo apt-get install -y libdbus-1-3 libxkbcommon-x11-0 libxcb-icccm4 \
-    libxcb-image0 libxcb-keysyms1 libxcb-randr0 libxcb-render-util0 \
-    libxcb-xinerama0 libxcb-xinput0 libxcb-xfixes0
-
-# Install xpra
-echo "Installing xpra"
-DISTRO=focal
-sudo apt-get install apt-transport-https software-properties-common
-sudo apt install ca-certificates
-sudo wget -O "/usr/share/keyrings/xpra-2022.gpg" https://xpra.org/xpra-2022.gpg
-sudo wget -O "/usr/share/keyrings/xpra-2018.gpg" https://xpra.org/xpra-2018.gpg
-sudo wget -O "/etc/apt/sources.list.d/xpra.list" https://xpra.org/repos/$DISTRO/xpra.list
-sudo apt-get update
-sudo apt-get -y install xpra xterm
-
-# Install pytorch and cellpose
-conda install pytorch torchvision torchaudio pytorch-cuda=11.6 -c pytorch -c nvidia
-pip install pyqtgraph PyQt5 numpy numba scipy natsort cellpose
-
-# Install github CLI for cloning private repo
-conda install -y gh --channel conda-forge
+curl -O https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
 
 # Login to github using tokens
 # gh auth login

src/client/README.md

@@ -6,23 +6,38 @@ The client of our data centric platform for microscopy imaging.
 
 ## How to use?
 
-### Installation
-This has been tested on Python versions 3.9, 3.10 and 3.11 on latest versions of Windows, Ubuntu and MacOS. In your dedicated environment run:
+### Installation 
+
+For installing dcp-client you will first need to clone the repo:
+
+```
+git clone https://github.com/HelmholtzAI-Consultants-Munich/data-centric-platform.git
+```
+
+Then navigate to the client directory:
 ```
-pip install dcp_client
+cd data-centric-platform/src/client
 ```
 
-### Installation for developers
-Before starting, make sure you have navigated to ```data-centric-platform/src/client```. All future steps expect you are in the client directory. This installation has been tested using a conda environment with python version 3.9 on a mac local machine. In your dedicated environment run:
+In your dedicated environment run:
 ```
 pip install -e .
 ```
 
+This installation has been thoroughly tested using a conda environment with python version 3.9, 3.10, 3.11 and 3.12 on a macOS local machine.
+
+
+
 #### Launch DCP client
 Make sure the server is already running, either locally or remotely. Then, depending on the configuration, simply run:
 ```
-dcp-client --mode local/remote
+dcp-client --mode local
+```
+or 
+```
+dcp-client --mode remote
 ```
+depending on whether your server is running locally or remotely.
 
 ## Want to know more?
 Visit our [documentation](https://data-centric-platform.readthedocs.io/en/latest/dcp_client_installation.html) for more information and a step by step guide on how to run the client.

src/client/dcp_client/gui/napari_window.py

@@ -4,6 +4,7 @@
 
 from qtpy.QtWidgets import QPushButton, QComboBox, QLabel, QGridLayout
 from qtpy.QtCore import Qt
+from qtpy.QtGui import QGuiApplication
 import napari
 import numpy as np
 
@@ -31,6 +32,8 @@ def __init__(self, app: Application) -> None:
         self.app = app
         self.setWindowTitle("napari viewer")
         self.setStyleSheet("background-color: #262930;")
+        #screen_size = QGuiApplication.primaryScreen().geometry()
+        #self.resize(int(screen_size.width()*0.9), int(screen_size.height()*0.8))
 
         # Load image and get corresponding segmentation filenames
         img = self.app.load_image()
@@ -39,6 +42,8 @@ def __init__(self, app: Application) -> None:
 
         # Set the viewer
         self.viewer = napari.Viewer(show=False)
+        self.viewer.window.add_plugin_dock_widget("napari-sam")
+
         self.viewer.add_image(img, name=get_path_stem(self.app.cur_selected_img))
         for seg_file in self.seg_files:
             self.viewer.add_labels(
@@ -238,6 +243,7 @@ def axis_changed(self, event) -> None:
         """
         self.active_mask_index = self.viewer.dims.current_step[0]
         masks = deepcopy(self.layer.data)
+
         # if user has switched to the instance mask
         if self.active_mask_index == 0:
             class_mask_with_contours = Compute4Mask.add_contour(masks[1], masks[0])
@@ -247,6 +253,7 @@ def axis_changed(self, event) -> None:
             ):
                 self.update_instance_mask(masks[0], masks[1])
             self.switch_to_instance_mask()
+
         # else if user has switched to the class mask
         elif self.active_mask_index == 1:
             if not check_equal_arrays(
@@ -260,6 +267,8 @@ def switch_to_instance_mask(self) -> None:
         Switch the application to the active mask mode by enabling 'paint_button', 'erase_button'
         and 'fill_button'.
         """
+
+        self.original_class_mask[self.cur_selected_seg] = deepcopy(self.layer.data[1])
         self.switch_controls("paint_button", True)
         self.switch_controls("erase_button", True)
         self.switch_controls("fill_button", True)
@@ -268,6 +277,8 @@ def switch_to_labels_mask(self) -> None:
         """
         Switch the application to non-active mask mode by enabling 'fill_button' and disabling 'paint_button' and 'erase_button'.
         """
+
+        self.original_instance_mask[self.cur_selected_seg] = deepcopy(self.layer.data[0])
         if self.cur_selected_seg in [layer.name for layer in self.viewer.layers]:
             self.viewer.layers[self.cur_selected_seg].mode = "pan_zoom"
         info_message_paint = (

src/client/dcp_client/utils/bentoml_model.py

@@ -88,4 +88,4 @@ def run_inference(self, data_path: str) -> List:
         :return: List of files not supported by the server if unsuccessful, otherwise returns None.
         """
         list_of_files_not_suported = self._run_inference(data_path)
-        return list_of_files_not_suported
+        return list_of_files_not_suported

src/client/dcp_client/utils/compute4mask.py

@@ -48,8 +48,8 @@ def get_contours(
                     contour[:, 0], contour[:, 1], contour_mask.shape
                 )
                 contour_mask[rr, cc] = instance_id
-            except:
-                print("Could not create contour for instance id", instance_id)
+            except Exception as error:
+                print("Could not create contour for instance id", instance_id, ". Error is :", error)
         return contour_mask
 
     @staticmethod

src/client/pyproject.toml

@@ -1,47 +1,3 @@
 [build-system]
-requires = ["setuptools>=61.0"]
+requires = ["setuptools>=42", "wheel"]
 build-backend = "setuptools.build_meta"
-
-[tool.setuptools]
-packages = ['dcp_client']
-
-[tool.setuptools.dynamic]
-dependencies = {file = ["requirements.txt"]}
-
-[project]
-name = "data-centric-platform-client"
-version = "0.1"
-requires-python = ">=3.9"
-description = "The client of the data centric platform for microscopy image segmentation"
-keywords = []
-classifiers = [
-    "Programming Language :: Python :: 3",
-    "Operating System :: OS Independent",
-]
-readme = "README.md"
-dynamic = ["dependencies"]
-authors = [
-  {name="Christina Bukas", email="christina.bukas@helmholtz-munich.de"},
-  {name="Helena Pelin", email="helena.pelin@helmholtz-munich.de"},
-  {name="Mariia Koren", email="mariia.koren@helmholtz-munich.de"},
-  {name="Marie Piraud", email="marie.piraud@helmholtz-munich.de"},
-]
-maintainers = [
-  {name="Christina Bukas", email="christina.bukas@helmholtz-munich.de"},
-  {name="Helena Pelin", email="helena.pelin@helmholtz-munich.de"}
-]
-
-[project.optional-dependencies]
-dev = [
-  "pytest>=7.4.3",
-  "pytest-qt>=4.2.0",
-  "sphinx",
-  "sphinx-rtd-theme"
-]
-
-[project.urls]
-repository = "https://github.com/HelmholtzAI-Consultants-Munich/data-centric-platform"
-documentation = "https://readthedocs.org/projects/data-centric-platform"
-
-[project.scripts]
-dcp-client = "dcp_client.main:main"