Imaging post on segmentation

[mrocklin]

Following on the loading and deconvolution posts (#26 #27) we would next like to perform image segmentation, and then get out some quantities for each region. What is the right way to do this?

@rxist525 @jakirkham @thewtex @jni

[jakirkham]

When it comes to segmentation, there are a lot of techniques one could employ. Here's a short list from ITK (requires some drilling down to get to actual implementations). Deep learning is also commonly employed here.

My temptation would be to treat segmentation more-or-less as a blackbox with a standard API. IOW one supplies an image and get out a label image (objects identified). Alternatively we could imagine this being two steps. Namely we get a mask of interesting portions of the image selected and then label those regions based on connectivity. Though we probably don't want to get much more granular than that. Users will already be more familiar with what segmentation algorithms work well on their data. This is just my opinion though. Others may have thoughts here. 🙂

We can certainly pick one from ITK to make things a bit more concrete. If you have thoughts on this, @rxist525, or if others have thoughts on this, that would be useful.

[jni]

Segmentation is probably a bit trickier than deconv... You probably want map_overlap with sufficient overlap to have "one full segment" in the overlap zone.

[jakirkham]

Yeah it depends on what our goal is here. With @rxist525's data, this would still be map_blocks as full frames fit comfortably in-memory. Generally this is not true though and map_overlap would be required (amongst other things). Thoughts on the intended scope here, @mrocklin?

[mrocklin]

So far we've been working on a particular dataset. I suggest that we just stick with that

…

On Wed, Aug 7, 2019 at 5:14 PM jakirkham ***@***.***> wrote: Yeah it depends on what are goal is here. With @rxist525 <https://github.com/rxist525>'s data, this would still be map_blocks as full frames fit comfortably in-memory. Generally this is not true though and map_overlap would be required (amongst other things). Thoughts on the intended scope here, @mrocklin <https://github.com/mrocklin>? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47?email_source=notifications&email_token=AACKZTHLRTJYHKFCUEHGMS3QDNQPNA5CNFSM4IJMITPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD32BSQI#issuecomment-519313729>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AACKZTBSWKZHOIZ6GGFM7CTQDNQPNANCNFSM4IJMITPA> .

[rxist525]

Lots of things to unpack here. Two main points to focus on are types of segmentation and scalability. While the type of data that you are currently working on will fit into memory, some of the operations (like tensor voting) would speed up geometrically with smaller volumes. On the other hand, we do have datasets (4D instead of 5D) that are several terabytes for a single volume.
https://www.biorxiv.org/content/10.1101/374140v1.supplementary-material
Perhaps a quick chat this week to discuss/brainstorm a few avenues to explore?

[jakirkham]

Previously @jni and I worked on extending connected components to larger than memory data. ( dask/dask-image#94 ) It was doable, but did require a few tricks and generated basically a module worth of code. Connected components was probably easier than segmentation will be as it was clearer (though not exactly straightforward) to resolve pixels near chunk boundaries. With segmentation I'd naively suspect resolving pixels near chunk boundaries is somewhat dependent on the segmentation algorithm used. While it could be interesting to explore this, I'm somewhat inclined to agree with @mrocklin that this is out of scope for a blogpost. It might be a reasonable PR though. Would you agree or do you have other thoughts?

[rxist525]

I agree.

…

On Thu, Aug 8, 2019 at 7:07 AM jakirkham ***@***.***> wrote: Previously @jni <https://github.com/jni> and I worked on extending connected components to larger than memory data. ( dask/dask-image#94 <dask/dask-image#94> ) It was doable, but did require a few tricks and generated basically a module worth of code. Connected components was probably easier than segmentation will be as it was clearer (though not exactly straightforward) to resolve pixels near chunk boundaries. With segmentation I'd naively suspect resolving pixels near chunk boundaries is somewhat dependent on the segmentation algorithm used. While it could be interesting to explore this, I'm somewhat inclined to agree with @mrocklin <https://github.com/mrocklin> that this is out of scope for a blogpost. It might be a reasonable PR though. Would you agree or do you have other thoughts? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#47?email_source=notifications&email_token=ACE3CWN4JVKIU3CKQGE6IV3QDQSA3A5CNFSM4IJMITPKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD33XGII#issuecomment-519533345>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ACE3CWJ53TF4AI6PRVBACRTQDQSA3ANCNFSM4IJMITPA> .

[mrocklin]

Perhaps a quick chat this week to discuss/brainstorm a few avenues to explore?

Sorry to miss this @rxist525 I'm up for chatting. I think that people like @jakirkham and @thewtex are probably more important to have in that conversation though.

[mrocklin]

@rxist525 also, I'm curious at what point you or someone you know might be interested in getting engaged in the development process here. Using the last post as a model, my guess is that embarrassingly parallel map_blocks + ITK workloads are straightforward enough that someone who is comfortable with Numpy and ITK could start engaging.

There would almost surely be problems, but we could work through those together.

From my perspective we're mostly bound by people who know what imaging routines to run being busy, rather than by people who understand Dask.

[jakirkham]

Yeah just to unpack @mrocklin's point a bit further and following up on a related conversation with @thewtex, I think one can get pretty far with a wrapper function that looks like this. Now one can use map_blocks with whatever one wants. We could do a similar thing with the Numba example at the end of the last blogpost to simplify this further and thus avoid even the use of map_blocks.

def itk_udf_wrapper(udf, img, udf_args, udf_kwargs):
    """ Apply user-defined function to a single chunk of data """
    import itk

    img = img[0, 0, ...]  # remove leading two length-one dimensions
    image = itk.image_view_from_array(img)   # Convert to ITK object

    op_result = udf(image, *udf_args, **udf_kwargs)  # Call ITK-based user function

    result = itk.array_from_image(op_result)  # Convert back to Numpy array
    result = result[None, None, ...]  # Add back the leading length-one dimensions

    return result

[jakirkham]

That said, the region properties side of the story may be a bit more interesting.

[rxist525]

@rxist525 also, I'm curious at what point you or someone you know might be interested in getting engaged in the development process here. Using the last post as a model, my guess is that embarrassingly parallel map_blocks + ITK workloads are straightforward enough that someone who is comfortable with Numpy and ITK could start engaging.

There would almost surely be problems, but we could work through those together.

From my perspective we're mostly bound by people who know what imaging routines to run being busy, rather than by people who understand Dask.

@mrocklin Great question - and good timing. I think we are just finished with our infrastructure setup that we can engage more seriously. We should setup a time to chat next week with my postdoc @ruanxt present. let's move this discussion to email?

[mrocklin]

I was chatting with @sofroniewn last week and he mentioned that it might also be interesting to try using a trained PyTorch model for segmentation. I would personally be fine with anything. This might also be a nice addition because this sequence would then have used all of scikit-image, ITK, and Torch to scale out image processing, which would be a nice Trifecta.

From my perspective neither scaling nor writing about this is the bottleneck. The bottleneck to doing this is just finding someone who can write up the imaging side of this, probably as a notebook, in such a way that I or someone else can then scale out easily without knowing too much about imaging.