Feature request: fewer densification steps

[mukundvarma]

Hey Saket,

Thanks for the python port of sctransform! I think it's a great method, and I'm really excited that you're making it accessible to more of the comp bio community with this port.

I've been looking at using pysctransform in my workflows, and I had a feature request. Not sure how easy/feasible it is, but would it be possible to either

1. replace methods on dense matrices with their sparse counterparts (e.g. here
2. process the sparse matrix in a chunked manner?

On large datasets (1M+ cells), keeping the dense matrix in memory requires either a prohibitively large and expensive machine, or the steps straight up fail.

I'd be happy to help if you already have some solutions in mind.

[saketkc]

Thanks Mukund, these are both great suggestions! I do have these in my list of todos. In short, the chunking idea is great and should be easy to implement. I will put create a branch and update it here soon with more thoughts.