How to use kerchunk with the ZarrPlugin in xpublish

[kthyng]

Have many people gone down this path? I have been able to get this to work with 1 ROMS model but not another. My impression from a bunch of digging around (but knowing next to nothing about zarr) is that when the ZarrPlugin acts on model output read in with xr.open_mfdataset() the encoding handling ends up different than when it acts on model output read in with xr.open_dataset() but pointing to a kerchunked json file of metadata about the model output.

In particular, the one model works because it doesn't have compression. The problem model output does and the filter and compressor encoding are not able to be serialized in the ZarrPlugin.
This issue is related: #223

[mpiannucci]

We do this at rps for ROMS and GFS data so far. If you'd like to see an example take a look at the xreds dataset provider.

These do not have compression though so we have not tackled that issue yet. I have hit the filter issue before though and it is on my list to track down eventually

[kthyng]

Thanks @mpiannucci. My model output is using filters and they weren't being converted into json correctly in the xpublish zarr plugin. I have a PR to address this here: #225

[wachsylon]

@kthyng instead of serving the dataset via the zarr plugin (including dask), one could also create a dataset plugin or even a APIRouter to host what is inside a fsspec.get_mapper(KERCHUNK_DATASET_SOURCE) . In this example, we add the new /kerchunk route which uses KERCHUNK_DATASET_SOURCE=dataset.encoding["source"].

• as the xpublish clients wouldnt see a kerchunk zarr but rather a normal zarr, we need to add a kv["zarr_consolidated_format"]=1 to the .zmetadata.
• it would be much shorter script if cache behavior what be more clear. right now, it calls gc.collect and sets options for caching but still some things are saved in cache no matter what.