ezSegmenter

[srivarra]

This is for internal use only; if you'd like to open an issue or request a new feature, please open a bug or enhancement issue

Instructions

This document should be filled out prior to embarking on any project that will take more than a couple hours to complete. The goal is to make sure that everyone is on the same page for the functionality and requirements of new features. Therefore, it's important that this is detailed enough to catch any misunderstandings beforehand. For larger projects, it can be useful to first give a high-level sketch, and then go back and fill in the details. For smaller ones, filling the entire thing out at once can be sufficient.

Relevant background

The goal of ezSegmenter (in MAUI) is to easily identify and create single object data from MIBI images. It focuses mainly on acellular (plaques, tangles) and cellular (microglia, astrocytic, dendritic cell projections) features.

We will be converting MAUI's ezSegmenter into Python, and integrating it with Ark.

The goal is to:

1. Simplify the process to go from cell and object segmentation to the cell table
2. Map out the MAUI GUI functionalities to Jupyter Lab notebooks
3. Merge Mesmer output to ezObject output at the mask level, reassigning cell labels before the marker count and cell table output

Design overview

These can be organized into essentially two notebooks:

1. Mask Merging Notebook
2. Segmentation Notebook

Mask Merging

1. Get the parent directory name for TIFFs to merge.
   1. Contains subfolders for the following: ezOutput, Manual Masks, MESMER Masks (already have that)
2. User inputs the masks they wish to merge with the cell masks
   1. The merging follows FIFO, First-in-First-out (queue)
3. User can change the overlap percentage value for each item in the mask merge queue.
4. Run the merge algorithm on all FOVs
   1. Log the parameters listed, the datetime of the merge step, name of objects merged, and directories for finding the merged files.
   2. At this point, masks should be viewable in Mantis.

Composite Builder and Object Segmentation Notebook

1. Input TIFFs can either be count-based TIFFs, or post-pixie TIFFS where each cluster is it's own image.
2. Set up the composite channel builder (user can supply nothing, and it will not create the channels.)
   1. Inputs include an argument for:
      1. addition
      2. subtraction, each lists of strings for the channels
         1. Set negative values to 0 at the end.
      3. the name to give the objects
      4. segmentation channel
      5. parameter information (kwargs perhaps to a function which isn't facing the user)
3. Make sure the check the channel names exist with validation utils
4. Similar to Rosetta, we run the step on a testing FOV, and then once the user is happy with their arguments, we apply the composite channel builder to the dataset.

Code mockup

Mask Merging

from queue import Queue
import xarray as xr
import loguru # best logging library in Python
import alpine as alp

def _mask_merger(merge_order: List[str], data: xr.DataArray, save_dir: pathlib.Path) -> None:
    merge_queue = Queue(merge_order)

    merged_mask: np.ndarray = np.zeros(shape = data.shape)

    while merge_queue:
        yield (channel := queue.deque())
        channel_data =  data.loc[..., channel]
        merged_mask += merge_func(channel_data)

        # log metadata at this step:
        merge_queue.dequeue()
        loguru.log_magic(file_name)
    
    alp.image_utils.save_image(merged_mask, save_dir + "merged")


def merge_masks(merge_order: List[str], merge_with: str, fovs_path: pathlib.Path) -> None:
    fovs = alp.io_utils.list_folders(fovs_path)
    for fov in fovs:
        fov_data = alp.load_utils.load_imgs_from_tree(fov)
        _mask_merger(merge_with = merge_with, merge_order = merge_order, data = fov_data, save_dir = fov / "merged_masks")
    
    create_mantis_dir(...)

Composite Builder and Segmentation

def setup_composite_masks(composite_fov_dir = pathlib.Path)
    composite_fov_dir.mkdir(parents=True, exist_ok=True)
    # Make any subfolders as well, ex.
    (composite_fov_dir / "subfolder").mkdir(parents=True, exist_ok=True)



def group_composite_builder(add_channels: List[str], subtract_channels: List[str], fovs_path: pathlib.Path, output_subdir: pathlib.Path) -> None:
    fovs = alp.io_utils.list_folders(fovs_path)
    for fov in fovs:
        fov_data = alp.load_utils.load_imgs_from_tree(fov)
        composite_fov = composite_mask_builder(add_channels, subtract_channels, fov_data)
        alp.image_utils.save_image(data = composite_fov, output_path = fov / output_subdir / "composite_fov.tiff")



def group_segmentation(fovs_path: pathlib.Path, object_names: List[str], output_subdir: pathlib.Path) -> None:
    fovs = alp.io_utils.list_folders(fovs_path)
    for fov in fovs:
        fov_data = alp.load_utils.load_imgs_from_tree(fov)
        segmented_fov = ez_segmentation(fov_data, object_names)
        for segmention, object_name in zip(segmented_fov, object_names):
            alp.image_utils.save_image(data = segmented_fov, output_path = fov / output_subdir / f"{object_name}.tiff")

        create_mantis_dir(...)

Required inputs

Provided a description of the required inputs for this project, including column names for dfs, dimensions for image data, prompts for user input, directory structure for loading data, etc

Inputs include:

1. FOV data in user_project/image_data.
2. segmentation/deepcell_output
3. ezOutput/*

Output files

Outputs include:

1. Merged objects
2. Segmentation objects
3. Log files which list the segmentation, and merging operations performed on the relevant channels.

data/
└── user_project/
    ├── image_data/
    │   ├── fov_0
    │   ├── fov_1
    │   ├── fov_2
    │   ├── ...
    │   └── fov_n
    ├── segmentation/
    │   ├── cell_table
    │   └── deepcell_output/
    │       ├── fov0_nuclear.tiff
    │       ├── fov0_whole_cell.tiff
    │       ├── ...
    │       ├── fovn_nuclear.tiff
    │       └── fovn_whole_cell.tiff
    └── ezOutput/
        ├── merged_masks/
        │   ├── object_1.tiff
        │   ├── object_2.tiff
        │   ├── ...
        │   └── object_m.tiff
        └── segemntation/
            ├── microglia.tiff
            ├── neropil.tiff
            ├── ...
            └── logfile.log

Timeline
Give a rough estimate for how long you think the project will take. In general, it's better to be too conservative rather than too optimistic.

• A couple days
• A week
• Multiple weeks. For large projects, make sure to agree on a plan that isn't just a single monster PR at the end.

Estimated date when a fully implemented version will be ready for review: 3/20

Estimated date when the finalized project will be merged in: 3/31

[srivarra]

@bryjcannon How does this look?

[bryjcannon]

@bryjcannon How does this look?

Looks good! A few notes.

• _mask_merger Ordering using queues makes sense - only thing we will need to figure out is how to add in and account for the MESMER cell table. Essentially while we run through the ez masks, we will start with merging with the original MESMER mask, remove any cells from the MESMER masks that have been merged, and use that new, mask-depleted MESMER mask, as the base to merge with the next set of ez masks.
• group_segmentation() I like the idea of being able to collectively create all of your segmentation masks (e.g. microglia, neuropil, large cells) through one notebook rather than separating each object into its own notebook. Unsure how to make this work within the flow of the UI though. May need a save settings feature that allows you to reset all the values and select again before committing to the final segmentation step.
• Will need a function for taking a Pixie tiff and breaking it down into X tiffs per FOV, where X = the individual number of colored pixel clusters. Should probably be a part of the composite and segmentation section.

Everything else looks great, thank you!

[srivarra]

@bryjcannon What is your vision for the UI? Is the Jupyter Notebook going to have interactive elements using ipywidgets?

[bryjcannon]

Using those would be fantastic, particularly for the segmentation parameter setting feature. It's the one thing I left out mainly because coding them up in the MATLAB version took a very long time. If you think it would be relatively easy to incorporate them here then yeah we should absolutely use them. Essentially the UI goes:

Composite & Segmentation notebook UI

1. Enter file paths
2. Build composite (if necessary): Display output in single FOV, alongside individual channels used to make composite
3. Name the object to segment
4. Choose channel to segment on (if pixie tiff run single-cluster channel generator)
5. Set parameters & generate example masking (if this can be done using widgets great, otherwise nw)
6. Segment across all FOVs, save values, logs

• Steps 3-6 in theory could be cycled within one notebook so you could segment multiple objects in the same notebook, but would require updating something in the background and in the UI denoting you've saved these values. If this would take too much time, creating multiple notebooks for each object is fine.

Merge notebook UI

1. Enter file paths for MESMER cell and ez object masks
2. Enter names of ez objects in order of merge status (first merged with cells first, second merged with cells second, etc.)
3. Merge over each FOV (no visualization needed before bringing the merged masks into Mantis)

• I mentioned this in the previous comment but we may need to tinker with the output at this step. As of now I imagine it will be masks of the objects merged with cells, plus a mask of any unmerged cells. These masks will then get run through the annotation & cell table construction steps. May need to create a merge cell tables feature for that.

[srivarra]

@bryjcannon I'd say lets get the behind-the-scenes functionality all set up, and once we know it's working correctly, we can try to create a fancy / fun UX with widgets and stuff. For the following bullet point below:

Will need a function for taking a Pixie tiff and breaking it down into X tiffs per FOV, where X = the individual number of colored pixel clusters. Should probably be a part of the composite and segmentation section.

Instead of breaking it apart to construct a new TIFF per cluster, there is some logic in

ark-analysis/src/ark/utils/plot_utils.py

Line 88 in fe65614

def plot_pixel_cell_cluster_overlay(img_xr, fovs, cluster_id_to_name_path, metacluster_colors,

which would allow us to store all masks on one TIFF, but then selectively subset clusters using a cluster_id_to_name file. Lemme know your thoughts on if this will still accomplish the same thing.

[bryjcannon]

Sounds good on both points.