Design discussion

[espg]

New xagg prototype up (#1 ) that gets at some of what I've been thinking about for aggregations. The lambda takes 6-to-7 minutes and $2-to-$3 to run a continental aggregation function on atl06 for an orbital cycle.

My hope is that the design pattern is someone portable and can be generalized. Here's the high-level overview of what happens:

1. Spacio-temporal metadata query. The atl06 example hits NASA CMR, but this is broadly compatible with anything that conforms to a STAC API
2. Definition of what to fetch (i.e., which columns); this is in a python function right now, but would probably be abstracted to a yml template. This is also where we might add in (for example) a reader specification (current example uses h5coro).
3. Definition of what aggregation to do. This is defined in the same python file, and could either be a script, function, or similar.
4. Definition of input / output grid. Input grid defines the workers; output grid is self explanatory. This is implemented using healpix for both, but could be generalized to arbitrary user defined grids later on.
5. What to write to. This is currently a stub (i.e., parquet files), but should be more grid like (i.e., to zarr or similar with partial spatial chunked writes)

In principle, we could (for minimal work) swap out ICESat-2 for anything in the CMR that has a data model such that we're reading h5 files with per observation lat/lon data-- that's a huge amount of science products already.

A few bigger picture items:

1. What do we write too? We currently use parquet/geoparquet, but Zarr seems like a promising candidate.
2. What do we visualize with? I'd love the ability to view this where data actually populated as it was written.
   • We had looked at longboard a while ago; it works with with geoparquet and is client side, which is great
     • My understanding is that it's based on deck.gl, which doesn't support non-Mercator projections, which is a deal breaker for the polar regions
     • How feasible is upstreaming multiple projection types?
   • Other options are zarr viewers
     • Requires fixed array output size (honestly a reasonable and workable restrictions)
     • Seem to have some support for healpix metadata (not required, but nice)
     • Some issues with only displaying global grid extent for the healpix example
3. Can we generalize the pattern to a common interface for both local and cloud deployment?
   • What's the cloud backend (cubed?), and how do we handle authentication
   • What's the local backend (vaex?)
   • How do we integrate with the existing ecosystem
4. Do we support custom input and output grids, or just custom output grids, or neither?

[maxrjones]

This is very cool!

What do we write too? We currently use parquet/geoparquet, but Zarr seems like a promising candidate

A primary factor is the expected downstream tools for working with the data. Zarr (likely via Icechunk) would be a good choice if you expect users to be working with the output using tools like Xarray, NumPy, PyTorch . If users would likely want to do statistics directly on the output in R or GIS software, Zarr probably isn't the best choice because support is unfortunately still lacking.

What do we visualize with? I'd love the ability to view this where data actually populated as it was written.

Xdggs has a nice integration with lonboard https://xdggs.readthedocs.io/en/latest/tutorials/h3.html#plotting. You're right about past limitations regarding projections. Just last month Kyle added globe view support which could work well if we go with dggs output + zarr (developmentseed/lonboard#908). It might require wiring those two pieces together and is again Python specific.

Can we generalize the pattern to a common interface for both local and cloud deployment?

I think so, especially if we use virtual zarr as an intermediary since the I/O optimizations would be decoupled from indexing/weight computations.

Do we support custom input and output grids, or just custom output grids, or neither?

we chatted about this a bit in slack, but I think targeting points -> DGGS is a good scope. The other decision is whether to optimize for polar dggs cells or global, since that could impact indexing approaches.

[espg]

The goal is to cast to a grid, so the target is xarray...so I suspect that means zarr.

Great to hear that deck.gl + lonboard added support for globe and orthographic views-- changes my view on viability for us! Still need to think a bit about if orthographic meets our needs, or if its worth spending cycles to push for a stereographic projection / view class. Are there any example of what the lonboard orthographic view / interface looks like in the wild that we could play with?