AnnData Conversion Notebook

[srivarra]

If you haven't already, please read through our contributing guidelines before opening your PR

What is the purpose of this PR?

Main Features

Adds a notebook for converting the cell table to per-FOV AnnData objects. By default, they get saved to data/<my-cohort>/anndata in the notebook. In addition, it briefly covers viewing components of the AnnData object, and creating data pipelines with a small application for constructing spatial neighbors with squidpy.

• Added the conversion class which creates a AnnData object with:
  • Renamed spatial coordinates (centroid-0 $\to$ centroid_y, centroid-1 $\to$ centroid_x), stored in obsm["spatial"].
  • All markers get stored in var_names / X.
  • All other columns get stored in obs.
  • Dropped settings.CELL_SIZE in favor of the generic "area" as a column name in obs, as we can have tables for many kinds of segmentations / observations (cell, fiber, ez_seg, etc...)
• Added a function to load all AnnData Zarr stores in a directory to an AnnCollection.
• Added a custom datapipe to iterate over the AnnCollection FOV by FOV.

Documentation

• Added a brief AnnData section for data_types.md with figures.
• Added a AnnData section for development.md.

Testing

• Adjusted test_utils.make_cell_table to have the expected ordering for Cell Tables. Added n_cells and n_markers to create different sized cell tables.
• Added a utility function to generate AnnData tables with various values of n_obs, and floating point and categorical column types. (test_utils.generate_anndata_table and test_utils.generate_anncollection)

How did you implement your changes

• Used Dask to work with converting large cell tables.
• Used TorchData to create the AnnData IterDataPipe. This allows us to implement more intricate data pipelines with shuffling, mapping, and filtering functionality, and construct data loaders with DataLoader2
• Adjusted tests which utilized test_utils.make_cell_table.

Remaining Issues

• Need a plan for structuring how we will write modules and functions for interacting with AnnData objects. We can try some stuff, and see what works best.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[alex-l-kong]

Beautiful stuff overall. I think once the lab gets familiar with AnnData/dask/zarr they'll find it very easy to use. Mostly structural comments.

[camisowers]

Notebook looks much cleaner!

[alex-l-kong]

Looks good, sorry for the late review

[srivarra]

@ngreenwald

[ngreenwald]

Does the embedded image show up correctly in the actual notebook? Not showing up here, could just be an issue with reviewnb

Reply via ReviewNB

[srivarra]

yeah, I think it's cause the image is being displayed with html, instead of markdown's native method.

Here's what it looks like when I open the notebook in Jupyter Lab.

[ngreenwald]

Looks awesome, just a couple small comments

[ngreenwald]

Looks good