+
+
+
+
+Kerchunk Cookbook
+
+
+
This Project Pythia Cookbook covers using the Kerchunk +library to access archival data formats as if they were +ARCO (Analysis-Ready-Cloud-Optimized) data.
+Motivation
+The Kerchunk library allows you to access chunked and compressed
+data formats (such as NetCDF3. HDF5, GRIB2, TIFF & FITS), many of
+which are the primary data formats for many data archives, as if
+they were in ARCO formats such as Zarr which allows for parallel,
+chunk-specific access. Instead of creating a new copy of the dataset
+in the Zarr spec/format, Kerchunk reads through the data archive
+and extracts the byte range and compression information of each
+chunk, then writes that information to a .json file (or alternate
+backends in future releases). For more details on how this process
+works please see this page on the
+Kerchunk docs).
+These summary files can then be combined to generated a Kerchunk
+reference for that dataset, which can be read via
+Zarr and
+Xarray.
Authors
+ +Much of the content of this cookbook was inspired by
+Martin Durant,
+the creator of Kerchunk and the
+Kerchunk documentation.
Contributors
+ +Structure
+This cookbook is broken up into two sections, +Foundations and Example Notebooks.
+Section 1 Foundations
+In the Foundations section we will demonstrate
+how to use Kerchunk to create reference sets
+from single file sources, as well as to create
+multi-file virtual datasets from collections of files.
Section 2 Generating Reference Files
+The notebooks in the Generating Reference Files section
+demonstrate how to use Kerchunk to create
+datasets for all the supported file formats.
+Kerchunk currently supports NetCDF3,
+NetCDF4/HDF5, GRIB2, TIFF (including CoG).
Section 3 Using Pre-Generated References
+The Pre-Generated References section contains notebooks demonstrating how to load existing references into Xarray and Xarray-Datatree, generated coordinates for GeoTiffs using xrefcoord and plotting using Hvplot Datashader.
Running the Notebooks
+You can either run the notebook using
+or on your local machine.
Running on Binder
+The simplest way to interact with a Jupyter Notebook is through +Binder, which enables the execution of a +Jupyter Book in the cloud. The details of how this works are not +important for now. All you need to know is how to launch a Pythia +Cookbooks chapter via Binder. Simply navigate your mouse to +the top right corner of the book chapter you are viewing and click +on the rocket ship icon and be sure to select +“launch Binder”. After a moment you should be presented with a +notebook that you can interact with. You’ll be able to execute +and even change the example programs. The code cells +have no output at first, until you execute them by pressing +Shift+Enter. Complete details on how to interact with +a live Jupyter notebook are described in Getting Started with +Jupyter.
+Running on Your Own Machine
+If you are interested in running this material locally on your computer, +you will need to follow this workflow:
+-
+
Install mambaforge/mamba
+Clone the
+https://github.com/ProjectPythia/kerchunk-cookbookrepository:++git clone https://github.com/ProjectPythia/kerchunk-cookbook.git +
+Move into the
+kerchunk-cookbookdirectory++cd kerchunk-cookbook +
+Create and activate your conda environment from the
+environment.ymlfile. +Note: In theenvironment.ymlfile, Kerchunk` is currently being installed from source as development is happening rapidly.++mamba env create -f environment.yml +mamba activate kerchunk-cookbook +
+Move into the
+notebooksdirectory and start up Jupyterlab++cd notebooks/ +jupyter lab +
+
+
+# Kerchunk Cookbook
+
+[](https://github.com/ProjectPythia/kerchunk-cookbook/actions/workflows/nightly-build.yaml)
+[](https://binder.projectpythia.org/v2/gh/ProjectPythia/kerchunk-cookbook/main?labpath=notebooks)
+[](https://zenodo.org/badge/latestdoi/588661659)
+
+This Project Pythia Cookbook covers using the [Kerchunk](https://fsspec.github.io/kerchunk/)
+library to access archival data formats as if they were
+ARCO (Analysis-Ready-Cloud-Optimized) data.
+
+## Motivation
+
+The `Kerchunk` library allows you to access chunked and compressed
+data formats (such as NetCDF3. HDF5, GRIB2, TIFF & FITS), many of
+which are the primary data formats for many data archives, as if
+they were in ARCO formats such as Zarr which allows for parallel,
+chunk-specific access. Instead of creating a new copy of the dataset
+in the Zarr spec/format, `Kerchunk` reads through the data archive
+and extracts the byte range and compression information of each
+chunk, then writes that information to a .json file (or alternate
+backends in future releases). For more details on how this process
+works please see this page on the
+[Kerchunk docs](https://fsspec.github.io/kerchunk/detail.html)).
+These summary files can then be combined to generated a `Kerchunk`
+reference for that dataset, which can be read via
+[Zarr](https://zarr.readthedocs.io) and
+[Xarray](https://docs.xarray.dev/en/stable/).
+
+## Authors
+
+[Raphael Hagen](https://github.com/norlandrhagen)
+
+Much of the content of this cookbook was inspired by
+[Martin Durant](https://github.com/martindurant),
+the creator of `Kerchunk` and the
+[Kerchunk documentation](https://fsspec.github.io/kerchunk/).
+
+### Contributors
+
+![\"Parquet](\"https://upload.wikimedia.org/wikipedia/commons/4/47/Apache_Parquet_logo.svg\"){kind=logo}
"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Store Kerchunk Reference Files as Parquet"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "In this notebook we will cover how to store Kerchunk references as Parquet files instead of json. For large reference datasets, using Parquet should have performance implications as the overall reference file size should be smaller and the memory overhead of combining the reference files should be lower. \n",
+ "\n",
+ "\n",
+ "This notebook builds upon the [Kerchunk Basics](notebooks/foundations/01_kerchunk_basics.ipynb), [Multi-File Datasets with Kerchunk](notebooks/foundations/02_kerchunk_multi_file.ipynb) and the [Kerchunk and Dask](notebooks/foundations/03_kerchunk_dask.ipynb) notebooks. \n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Recommended | IO/Visualization |\n",
+ "| [Intro to Dask](https://tutorial.dask.org/00_overview.html) | Required | Parallel Processing |\n",
+ "\n",
+ "- **Time to learn**: 30 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports\n",
+ "In addition to the previous imports we used throughout the tutorial, we are adding a few imports:\n",
+ "- `LazyReferenceMapper` and `ReferenceFileSystem` from `fsspec.implementations.reference` for lazy Parquet.\n",
+ "\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "import os\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import xarray as xr\n",
+ "from distributed import Client\n",
+ "from fsspec.implementations.reference import LazyReferenceMapper, ReferenceFileSystem\n",
+ "from kerchunk.combine import MultiZarrToZarr\n",
+ "from kerchunk.hdf import SingleHdf5ToZarr"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setting up the `Dask` Client\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create Input File List\n",
+ "\n",
+ "Here we are using `fsspec's` glob functionality along with the *`*`* wildcard operator and some string slicing to grab a list of NetCDF files from a `s3` `fsspec` filesystem. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "files_paths = fs_read.glob(\"s3://smn-ar-wrf/DATA/WRF/DET/2022/12/31/12/*\")\n",
+ "\n",
+ "# Here we prepend the prefix 's3://', which points to AWS.\n",
+ "file_pattern = sorted([\"s3://\" + f for f in files_paths])\n",
+ "\n",
+ "# Grab the first seven files to speed up example.\n",
+ "file_pattern = file_pattern[0:2]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Generate Lazy References\n",
+ "Below we will create a `fsspec` filesystem to read the references from `s3` and create a function to generate dask delayed tasks."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use Kerchunk's `SingleHdf5ToZarr` method to create a `Kerchunk`\n",
+ "# index from a NetCDF file.\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "so = dict(mode=\"rb\", anon=True, default_fill_cache=False, default_cache_type=\"first\")\n",
+ "\n",
+ "\n",
+ "def generate_json_reference(fil):\n",
+ " with fs_read.open(fil, **so) as infile:\n",
+ " h5chunks = SingleHdf5ToZarr(infile, fil, inline_threshold=300)\n",
+ " return h5chunks.translate() # outf\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil) for fil in file_pattern]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Start the Dask Processing\n",
+ "To view the processing you can view it in real-time on the Dask Dashboard. ex: http://127.0.0.1:8787/status"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "single_refs = dask.compute(tasks)[0]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "len(single_refs)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Combine In-Memory References with MultiZarrToZarr\n",
+ "This section will look notably different than the previous examples that have written to `.json`.\n",
+ "\n",
+ "In the following code block we are:\n",
+ "- Creating an `fsspec` filesystem.\n",
+ "- Create a empty `parquet` file to write to. \n",
+ "- Creating an `fsspec` `LazyReferenceMapper` to pass into `MultiZarrToZarr`\n",
+ "- Building a `MultiZarrToZarr` object of the combined references.\n",
+ "- Calling `.flush()` on our LazyReferenceMapper, to write the combined reference to our `parquet` file."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "fs = fsspec.filesystem(\"file\")\n",
+ "\n",
+ "if os.path.exists(\"combined.parq\"):\n",
+ " import shutil\n",
+ "\n",
+ " shutil.rmtree(\"combined.parq\")\n",
+ "os.makedirs(\"combined.parq\")\n",
+ "\n",
+ "out = LazyReferenceMapper.create(root=\"combined.parq\", fs=fs, record_size=1000)\n",
+ "\n",
+ "mzz = MultiZarrToZarr(\n",
+ " single_refs,\n",
+ " remote_protocol=\"s3\",\n",
+ " concat_dims=[\"time\"],\n",
+ " identical_dims=[\"y\", \"x\"],\n",
+ " remote_options={\"anon\": True},\n",
+ " out=out,\n",
+ ").translate()\n",
+ "\n",
+ "out.flush()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Shutdown the Dask cluster"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client.shutdown()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Load kerchunked dataset\n",
+ "\n",
+ "Next we initiate a `fsspec` `ReferenceFileSystem`.\n",
+ "We need to pass:\n",
+ "- The name of the parquet store\n",
+ "- The remote protocol (This is the protocol of the input file urls)\n",
+ "- The target protocol (`file` since we saved our parquet store locally)."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "fs = ReferenceFileSystem(\n",
+ " \"combined.parq\", remote_protocol=\"s3\", target_protocol=\"file\", lazy=True\n",
+ ")\n",
+ "ds = xr.open_dataset(\n",
+ " fs.get_mapper(), engine=\"zarr\", backend_kwargs={\"consolidated\": False}\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ds"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "kerchunk-cookbook-dev",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/_preview/58/_sources/notebooks/advanced/appending.ipynb b/_preview/58/_sources/notebooks/advanced/appending.ipynb
new file mode 100644
index 0000000..95be830
--- /dev/null
+++ b/_preview/58/_sources/notebooks/advanced/appending.ipynb
@@ -0,0 +1,332 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Appending to Kerchunk references\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ "\n",
+ "In this tutorial we'll show how to append to a pre-existing Kerchunk reference. We'll use the same datasets as in the [NetCDF reference generation](../generating_references/NetCDF.ipynb) example. \n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Multi-File Datasets with Kerchunk](../case_studies/ARG_Weather.ipynb) | Required | IO/Visualization |\n",
+ "\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "from tempfile import TemporaryDirectory\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import ujson\n",
+ "import xarray as xr\n",
+ "from distributed import Client\n",
+ "from kerchunk.combine import MultiZarrToZarr\n",
+ "from kerchunk.hdf import SingleHdf5ToZarr"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create Input File List\n",
+ "\n",
+ "Here we are using `fsspec's` glob functionality along with the *`*`* wildcard operator and some string slicing to grab a list of NetCDF files from a `s3` `fsspec` filesystem. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "files_paths = fs_read.glob(\n",
+ " \"s3://smn-ar-wrf/DATA/WRF/DET/2022/12/31/12/WRFDETAR_01H_20221231_12_*\"\n",
+ ")\n",
+ "\n",
+ "# Here we prepend the prefix 's3://', which points to AWS.\n",
+ "file_pattern = sorted([\"s3://\" + f for f in files_paths])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This dictionary will be passed as kwargs to `fsspec`. For more details, check out the\n",
+ "# `foundations/kerchunk_basics` notebook.\n",
+ "so = dict(mode=\"rb\", anon=True, default_fill_cache=False, default_cache_type=\"first\")\n",
+ "\n",
+ "# We are creating a temporary directory to store the .json reference files\n",
+ "# Alternately, you could write these to cloud storage.\n",
+ "td = TemporaryDirectory()\n",
+ "temp_dir = td.name"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Start a Dask Client\n",
+ "\n",
+ "To parallelize the creation of our reference files, we will use `Dask`. For a detailed guide on how to use Dask and Kerchunk, see the Foundations notebook: [Kerchunk and Dask](../foundations/kerchunk_dask).\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create a `Kerchunk` reference file for the first 24 hours"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "first_24_hrs = file_pattern[:24]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use Kerchunk's `SingleHdf5ToZarr` method to create a `Kerchunk` index from\n",
+ "# a NetCDF file.\n",
+ "\n",
+ "\n",
+ "def generate_json_reference(fil, output_dir: str):\n",
+ " with fs_read.open(fil, **so) as infile:\n",
+ " h5chunks = SingleHdf5ToZarr(infile, fil, inline_threshold=300)\n",
+ " fname = fil.split(\"/\")[-1].strip(\".nc\")\n",
+ " outf = f\"{output_dir}/{fname}.json\"\n",
+ " with open(outf, \"wb\") as f:\n",
+ " f.write(ujson.dumps(h5chunks.translate()).encode())\n",
+ " return outf\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil, temp_dir) for fil in first_24_hrs]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start parallel processing\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "dask.compute(tasks)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Combine .json `Kerchunk` reference files and write a combined `Kerchunk` index\n",
+ "\n",
+ "In the following cell, we are combining all the `.json` reference files that were generated above into a single reference file and writing that file to disk."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create a list of reference json files\n",
+ "output_files = [\n",
+ " f\"{temp_dir}/{f.strip('.nc').split('/')[-1]}.json\" for f in first_24_hrs\n",
+ "]\n",
+ "\n",
+ "# combine individual references into single consolidated reference\n",
+ "mzz = MultiZarrToZarr(\n",
+ " output_files,\n",
+ " concat_dims=[\"time\"],\n",
+ " identical_dims=[\"y\", \"x\"],\n",
+ " remote_protocol=\"s3\",\n",
+ " remote_options={\"anon\": True},\n",
+ " coo_map={\"time\": \"cf:time\"},\n",
+ ")\n",
+ "# save translate reference in memory for later visualization\n",
+ "multi_kerchunk = mzz.translate()\n",
+ "\n",
+ "# Write kerchunk .json record.\n",
+ "output_fname = \"ARG_combined.json\"\n",
+ "with open(f\"{output_fname}\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(multi_kerchunk).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Append references for the next 24 hours\n",
+ "\n",
+ "We'll now append the references for the next 24 hours. First, we create an individual temporary reference file for each input data file. Then,\n",
+ "we load the original references and append the new references."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# First generate the individual reference files to be appended\n",
+ "\n",
+ "second_24_hrs = file_pattern[24:48]\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil, temp_dir) for fil in second_24_hrs]\n",
+ "\n",
+ "# Generate reference files for the individual NetCDF files\n",
+ "dask.compute(tasks)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Load the original references\n",
+ "fs_local = fsspec.filesystem(\"file\")\n",
+ "archive = ujson.load(fs_local.open(output_fname))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create a list of individual reference files to append to the combined reference\n",
+ "output_files = [\n",
+ " f\"{temp_dir}/{f.strip('.nc').split('/')[-1]}.json\" for f in second_24_hrs\n",
+ "]\n",
+ "\n",
+ "# Append to the existing reference file\n",
+ "mzz = MultiZarrToZarr.append(\n",
+ " output_files,\n",
+ " original_refs=archive,\n",
+ " concat_dims=[\"time\"],\n",
+ " identical_dims=[\"y\", \"x\"],\n",
+ " remote_protocol=\"s3\",\n",
+ " remote_options={\"anon\": True},\n",
+ " coo_map={\"time\": \"cf:time\"},\n",
+ ")\n",
+ "\n",
+ "multi_kerchunk = mzz.translate()\n",
+ "\n",
+ "# Write kerchunk .json record.\n",
+ "output_fname = \"ARG_combined.json\"\n",
+ "with open(f\"{output_fname}\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(multi_kerchunk).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Opening Reference Dataset with Fsspec and Xarray\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "storage_options = {\n",
+ " \"remote_protocol\": \"s3\",\n",
+ " \"skip_instance_cache\": True,\n",
+ "} # options passed to fsspec\n",
+ "open_dataset_options = {\"chunks\": {}} # opens passed to xarray\n",
+ "\n",
+ "ds = xr.open_dataset(\n",
+ " \"ARG_combined.json\",\n",
+ " engine=\"kerchunk\",\n",
+ " storage_options=storage_options,\n",
+ " open_dataset_options=open_dataset_options,\n",
+ ")\n",
+ "\n",
+ "ds"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "kerchunk-cookbook",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/_preview/58/_sources/notebooks/foundations/01_kerchunk_basics.ipynb b/_preview/58/_sources/notebooks/foundations/01_kerchunk_basics.ipynb
new file mode 100644
index 0000000..67b72db
--- /dev/null
+++ b/_preview/58/_sources/notebooks/foundations/01_kerchunk_basics.ipynb
@@ -0,0 +1,219 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![\"Kerchunk](\"../../thumbnail.png\")
\n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Kerchunk Basics"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ "\n",
+ "This notebook is intended as an introduction to using `Kerchunk`.\n",
+ "In this tutorial we will:\n",
+ "- Scan a single NetCDF4/HDF5 file to create a `Kerchunk` virtual dataset\n",
+ "- Learn how to use the output using `Xarray` and `fsspec`.\n",
+ "\n",
+ "While this notebook only examines using `Kerchunk` on a single NetCDF file, `Kerchunk` can be used to create virtual `Zarr` datasets from collections of many input files. In the following notebook, we will demonstrate this. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Helpful | Basic features |\n",
+ "\n",
+ "- **Time to learn**: 60 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports\n",
+ "\n",
+ "Here we will import a few `Python` libraries to help with our data processing. \n",
+ "- `fsspec` will be used to read remote and local filesystems. \n",
+ "- `kerchunk.hdf` will be used to read a NetCDF file and create a `Kerchunk` reference set.\n",
+ "- `ujson` for writing the `Kerchunk` output to the `.json` file format.\n",
+ "- `Xarray` for examining out output dataset\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import fsspec\n",
+ "import kerchunk.hdf\n",
+ "import ujson\n",
+ "import xarray as xr"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Define kwargs for `fsspec`\n",
+ "\n",
+ "In the dictionary definition in the next cell, we are passing options to [`fsspec.open`](https://filesystem-spec.readthedocs.io/en/latest/api.html?highlight=.open#fsspec.open). Any additional kwargs passed in this dictionary through `fsspec.open` will pass as kwargs to the file system, in our case `s3`. The API docs for the `s3fs` filesystem spec can be found [here](https://s3fs.readthedocs.io/en/latest/api.html).\n",
+ "\n",
+ "In this example we are passing a few kwargs. In short they are:\n",
+ "- `anon=True`: This is a `s3fs` kwarg that specifies you are not passing any connection credentials and are connecting to a public bucket.\n",
+ "- `default_fill_cache=False`: `s3fs` kwarg that avoids caching in between chunks of files. This may lower memory usage when reading large files.\n",
+ "- `default_cache_type=\"first\"`: `fsspec` kwarg that specifies the caching strategy used by `fsspec`. In this case, `first` caches the first block of a file only.\n",
+ "\n",
+ "Don't worry too much about the details here; the cache options are those that have typically proven efficient for HDF5 files. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "so = dict(anon=True, default_fill_cache=False, default_cache_type=\"first\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Parse a single NetCDF file with kerchunk\n",
+ "\n",
+ "Below we will access a NetCDF file stored on the AWS cloud. This dataset is a single time slice of a climate downscaled product for Alaska.\n",
+ "\n",
+ "The steps in the cell below are as follows:\n",
+ "1. Define the url that points to the `NetCDF` file we want to process\n",
+ "1. Use `fsspec.open` along with the dictionary of arguments we created (`so`) to open the URL pointing to the NetCDF file.\n",
+ "1. Use `kerchunk.hdf.SingleHdf5ToZarr` method to read through the `NetCDF` file and extract the byte ranges, compression information and metadata.\n",
+ "1. Use `Kerchunk's` `.translate` method on the output from the `kerchunk.hdf.SingleHdf5ToZarr` to translate content of the NetCDF file into the `Zarr` format.\n",
+ "1. Create a `.json` file named `single_file_kerchunk.json` and write the dataset information to disk.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Input URL to dataset. Note this is a netcdf file stored on s3 (cloud dataset).\n",
+ "url = \"s3://wrf-se-ak-ar5/ccsm/rcp85/daily/2060/WRFDS_2060-01-01.nc\"\n",
+ "\n",
+ "# Uses kerchunk to scan through the netcdf file to create kerchunk mapping and\n",
+ "# then save output as .json.\n",
+ "# Note: In this example, we write the kerchunk output to a .json file.\n",
+ "# You could also keep this information in memory and pass it to fsspec\n",
+ "with fsspec.open(url, **so) as inf:\n",
+ " h5chunks = kerchunk.hdf.SingleHdf5ToZarr(inf, url, inline_threshold=100)\n",
+ " h5chunks.translate()\n",
+ " with open(\"single_file_kerchunk.json\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(h5chunks.translate()).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Load `Kerchunk` Reference File\n",
+ "\n",
+ "In the section below we will use `fsspec.filesystem` along with the `Kerchunk` `.json` reference file to open the `NetCDF` file as if it were a `Zarr` dataset."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# use fsspec to create filesystem from .json reference file\n",
+ "fs = fsspec.filesystem(\n",
+ " \"reference\",\n",
+ " fo=\"single_file_kerchunk.json\",\n",
+ " remote_protocol=\"s3\",\n",
+ " remote_options=dict(anon=True),\n",
+ " skip_instance_cache=True,\n",
+ ")\n",
+ "\n",
+ "# load kerchunked dataset with xarray\n",
+ "ds = xr.open_dataset(\n",
+ " fs.get_mapper(\"\"), engine=\"zarr\", backend_kwargs={\"consolidated\": False}\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Plot Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ds.TMAX.plot()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Note that the original .nc file size here is 16.8MB, and the created JSON is 26.5kB. These files also tend to compress very well. As you can see, it the JSON can be written anywhere, and gives us access to the underlying data, reading only the chunks we need from remote without downloading the whole file."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "kerchunk-cookbook-dev",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "89095a95fbc59e1db286735bee0073a08e46abd63daa66f53634eb5c8cc2192a"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/_preview/58/_sources/notebooks/foundations/02_kerchunk_multi_file.ipynb b/_preview/58/_sources/notebooks/foundations/02_kerchunk_multi_file.ipynb
new file mode 100644
index 0000000..5942aa7
--- /dev/null
+++ b/_preview/58/_sources/notebooks/foundations/02_kerchunk_multi_file.ipynb
@@ -0,0 +1,407 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![\"Dask](\"../images/dask.png\")
"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Kerchunk and Dask"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "In this notebook we will cover:\n",
+ "\n",
+ "1. How to parallelize the creation of `Kerchunk` reference files using the `Dask` library.\n",
+ "1. How to scale up the creation of large `Kerchunk` datasets using `Dask`. \n",
+ "\n",
+ "This notebook builds upon the [Kerchunk Basics](./kerchunk_basics.ipynb) and the [Multi-File Datasets with Kerchunk](../foundations/kerchunk_multi_file.ipynb) notebooks. A basic understanding of `Dask` will be helpful, but is not required. This notebook is not intended as a tutorial for using `Dask`, but will show how to use `Dask` to greatly speedup the the generation of `Kerchunk` reference files. \n",
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Recommended | IO/Visualization |\n",
+ "| [Intro to Dask](https://tutorial.dask.org/00_overview.html) | Recommended | Parallel Processing |\n",
+ "\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dask and Parallel Processing\n",
+ "\n",
+ "Dask is a `Python` library for parallel computing. It plays well with `Xarray`, but can be used in many ways across the `Python` ecosystem. This notebook is not intended to be a guide for how to use `Dask`, but just an example of how to use `Dask` to parallelize some `Kerchunk` functionality. \n",
+ "\n",
+ "In the previous notebook [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file), we were looking at daily downscaled climate data over South-Eastern Alaska. In our function named `generate_json_reference`, we were looping over, one at a time, the input NetCDF4 files and using `Kerchunk's` `SingleHdf5ToZarr` method to create a `Kerchunk` index for each file.\n",
+ "\n",
+ "With `Dask`, we can call `SingleHdf5ToZarr` in parallel, which allows us to create multiple `Kerchunk` reference files at the same time.\n",
+ "\n",
+ "Further on in this notebook, we will show how using `Dask` can greatly speed-up the process of creating a virtual dataset using `Kerchunk`\n",
+ "\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Setting up the `Dask` Client\n",
+ "\n",
+ "In the code below, we are importing `Dask Disributed` and creating a `client`. This is the start of our parallel `Kerchunk` data processing. We are passing the argument `n_workers=8`. This will inform the `Dask` client on some of the resource limitations. \n",
+ "\n",
+ "Note: Depending on if you are running on a small machine such as a laptop or a larger compute hub, these resources could be tuned to improve performance. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import logging\n",
+ "\n",
+ "from distributed import Client\n",
+ "\n",
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Binder Specific Setup\n",
+ "\n",
+ "If you are running this tutorial on `Binder`, the configuration may look slightly different. \n",
+ "\n",
+ "Once you start the client, some information should be returned to you as well as a button that says:\n",
+ "\n",
+ "**Launch Dashboard in JupyterLab**\n",
+ "\n",
+ "\n",
+ "\n",
+ "![](\"../images/client_binder.png\")
\n",
+ " \n",
+ "\n",
+ "Once you click that button, multiple windows of the `Dask` dashboard should open up. \n",
+ "\n",
+ "\n",
+ "![](\"../images/binder_client2.png\")
\n",
+ " \n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Building off of our Previous Work\n",
+ "In the next section, we will re-use some of the code from [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) notebook. However, we will modify it slightly to make it compatible with `Dask`.\n",
+ "\n",
+ "The following two cells should look the same as before. As a reminder we are importing the required libraries, using `fsspec` to create a list of our input files and setting up some kwargs for `fsspec` to use. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from tempfile import TemporaryDirectory\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import ujson\n",
+ "from kerchunk.hdf import SingleHdf5ToZarr"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading and writing\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "# Retrieve list of available days in archive for the year 2060.\n",
+ "files_paths = fs_read.glob(\"s3://wrf-se-ak-ar5/ccsm/rcp85/daily/2060/*\")\n",
+ "\n",
+ "# Here we prepend the prefix 's3://', which points to AWS.\n",
+ "file_pattern = sorted([\"s3://\" + f for f in files_paths])\n",
+ "\n",
+ "# Define kwargs for `fsspec`\n",
+ "so = dict(mode=\"rb\", anon=True, default_fill_cache=False, default_cache_type=\"first\")\n",
+ "\n",
+ "\n",
+ "# We are creating a temporary directory to store the .json reference files\n",
+ "# Alternately, you could write these to cloud storage.\n",
+ "td = TemporaryDirectory()\n",
+ "temp_dir = td.name\n",
+ "temp_dir"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Subset the Data\n",
+ "To speed up our example, lets take a subset of the year of data. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "file_pattern = file_pattern[0:40]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dask Specific Changes\n",
+ "\n",
+ "Here is the section of code that will change. Instead of iterating through each input file and using `generate_json_reference` to create the `Kerchunk` reference files, we are iterating through our input file list and creating `Dask Delayed Objects`. It is not super important to understand this, but a `Dask Delayed Object` is lazy, meaning it is not computed eagerly. Once we have iterated through all our input files, we end up with a list of `Dask Delayed Objects`. \n",
+ "\n",
+ "When we are ready, we can call `dask.compute` on this list of delayed objects to create `Kerchunk` reference files in parallel. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use Kerchunk's `SingleHdf5ToZarr` method to create a `Kerchunk` index\n",
+ "# from a NetCDF file.\n",
+ "\n",
+ "\n",
+ "def generate_json_reference(fil, temp_dir: str):\n",
+ " with fs_read.open(fil, **so) as infile:\n",
+ " h5chunks = SingleHdf5ToZarr(infile, fil, inline_threshold=300)\n",
+ " fname = fil.split(\"/\")[-1].strip(\".nc\")\n",
+ " outf = f\"{temp_dir}/{fname}.json\"\n",
+ " with open(outf, \"wb\") as f:\n",
+ " f.write(ujson.dumps(h5chunks.translate()).encode())\n",
+ " return outf\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil, temp_dir) for fil in file_pattern]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Dask Task Graph\n",
+ "\n",
+ "Once you call `dask.compute` it can be hard to understand what is happening and how far along the process is at any time. Fortunately, `Dask` has a built in dashboard to help visualize your progress.\n",
+ "\n",
+ "### Running this notebook locally\n",
+ "When you first initialized the `Dask` `client` earlier on, it should have returned some information including an address to the `dashboard`. For example: `http://127.0.0.1:8787/status`\n",
+ "\n",
+ "\n",
+ "By navigating to this address, you should a `Dask` dashboard that looks something like this.\n",
+ "\n",
+ "![](\"../images/empty_dashboard.png\"){kind=spacer}
\n",
+ " \n",
+ "![](\"../images/dashboard.png\")
\n",
+ "\n",
+ "![](\"../images/task_stream_full.png\")
\n",
+ "\n",
+ "![\"HRRR](\"../images/GRIB2.png\")
\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9eced552",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "Within this notebook, we will cover:\n",
+ "\n",
+ "1. Generating a list of GRIB2 files on a remote filesystem using `fsspec`\n",
+ "1. How to create reference files of GRIB2 files using ``Kerchunk`` \n",
+ "1. Combining multiple `Kerchunk` reference files using `MultiZarrToZarr`\n",
+ "\n",
+ "This notebook shares many similarities with the [Multi-File Datasets with Kerchunk](../foundations/kerchunk_multi_file.ipynb) and the [NetCDF/HDF5 Argentinian Weather Dataset Case Study](../case_studies/ARG_Weather.ipynb), however this case studies examines another data format and uses `kerchunk.scan_grib` to create reference files. \n",
+ "\n",
+ "This notebook borrows heavily from this [GIST](https://gist.github.com/peterm790/92eb1df3d58ba41d3411f8a840be2452) created by [Peter Marsh](https://gist.github.com/peterm790). \n",
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Kerchunk and Dask](../foundations/kerchunk_dask) | Required | Core |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Required | IO/Visualization |\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4043365b",
+ "metadata": {},
+ "source": [
+ "## Motivation\n",
+ "\n",
+ "`Kerchunk` supports multiple input file formats. One of these is `GRIB2(GRIdded Information in Binary form)`, which is a binary file format primary used in meteorology and weather datasets. Similar to NetCDF/HDF5, GRIB2 does not support efficient, parallel access. Using `Kerchunk`, we can read this legacy format as if it were an ARCO (Analysis-Ready, Cloud-Optimized) data format such as Zarr. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e3469ac4",
+ "metadata": {},
+ "source": [
+ "## About the Dataset\n",
+ "\n",
+ "The `HRRR` is a NOAA real-time 3-km resolution, hourly updated, cloud-resolving, convection-allowing atmospheric model, initialized by 3km grids with 3km radar assimilation. Radar data is assimilated in the HRRR every 15 min over a 1-h period adding further detail to that provided by the hourly data assimilation from the 13km radar-enhanced Rapid Refresh.\n",
+ "NOAA releases a copy of this dataset via the AWS Registry of Open Data. \n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "0e7ce9e2",
+ "metadata": {},
+ "source": [
+ "## Flags\n",
+ "In the section below, set the `subset` flag to be `True` (default) or `False` depending if you want this notebook to process the full file list. If set to `True`, then a subset of the file list will be processed (Recommended)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "5ec42e0d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "subset_flag = True"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f8d0f8f1",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "706c1b95",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import glob\n",
+ "import logging\n",
+ "from tempfile import TemporaryDirectory\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import ujson\n",
+ "from distributed import Client\n",
+ "from kerchunk.combine import MultiZarrToZarr\n",
+ "from kerchunk.grib2 import scan_grib"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "858399ce",
+ "metadata": {},
+ "source": [
+ "## Create Input File List\n",
+ "\n",
+ "Here we create `fsspec` files-systems for reading remote files and writing local reference files.\n",
+ "Next we are using `fsspec.glob` to retrieve a list of file paths and appending the `s3://` prefix to them."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "06fb2af3",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading and writing\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "# retrieve list of available days in archive\n",
+ "days_available = fs_read.glob(\"s3://noaa-hrrr-bdp-pds/hrrr.*\")\n",
+ "\n",
+ "# Read HRRR GRIB2 files from April 19, 2023\n",
+ "files = fs_read.glob(\"s3://noaa-hrrr-bdp-pds/hrrr.20230419/conus/*wrfsfcf01.grib2\")\n",
+ "\n",
+ "# Append s3 prefix for filelist\n",
+ "files = sorted([\"s3://\" + f for f in files])\n",
+ "\n",
+ "# If the subset_flag == True (default), the list of input files will be subset to\n",
+ "# speed up the processing\n",
+ "if subset_flag:\n",
+ " files = files[0:2]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "3bec2881",
+ "metadata": {},
+ "source": [
+ "## Start a Dask Client\n",
+ "\n",
+ "To parallelize the creation of our reference files, we will use `Dask`. For a detailed guide on how to use Dask and Kerchunk, see the Foundations notebook: [Kerchunk and Dask](../foundations/kerchunk_dask).\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "d99ee8ce",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "5ec53627",
+ "metadata": {},
+ "source": [
+ "## Iterate through list of files and create `Kerchunk` indices as `.json` reference files\n",
+ "\n",
+ "Each input GRIB2 file contains multiple \"messages\", each a measure of some variable on a grid, but with grid dimensions not necessarily compatible with one-another. The filter we create in the first line selects only certain types of messages, and indicated that heightAboveGround will be a coordinate of interest.\n",
+ "\n",
+ "We also write a separate JSON for each of the selected message, since these are the basic component data sets (see the loop over `out`).\n",
+ "\n",
+ "**Note**: `scan_grib` does not require a filter and will happily create a reference file for each available grib message. However when combining the grib messages using `MultiZarrToZarr` it is necessary for the messages to share a coordinate system. Thus to make our lives easier and ensure all reference outputs from `scan_grib` share a coordinate system we pass a filter argument.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "aa89197b",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "afilter = {\"typeOfLevel\": \"heightAboveGround\", \"level\": [2, 10]}\n",
+ "so = {\"anon\": True}\n",
+ "\n",
+ "# We are creating a temporary directory to store the .json reference files\n",
+ "# Alternately, you could write these to cloud storage.\n",
+ "td = TemporaryDirectory()\n",
+ "temp_dir = td.name\n",
+ "temp_dir"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "42d45880",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def make_json_name(\n",
+ " file_url, message_number\n",
+ "): # create a unique name for each reference file\n",
+ " date = file_url.split(\"/\")[3].split(\".\")[1]\n",
+ " name = file_url.split(\"/\")[5].split(\".\")[1:3]\n",
+ " return f\"{temp_dir}/{date}_{name[0]}_{name[1]}_message{message_number}.json\"\n",
+ "\n",
+ "\n",
+ "def gen_json(file_url):\n",
+ " out = scan_grib(\n",
+ " file_url, storage_options=so, inline_threshold=100, filter=afilter\n",
+ " ) # create the reference using scan_grib\n",
+ " for i, message in enumerate(\n",
+ " out\n",
+ " ): # scan_grib outputs a list containing one reference per grib message\n",
+ " out_file_name = make_json_name(file_url, i) # get name\n",
+ " with open(out_file_name, \"w\") as f:\n",
+ " f.write(ujson.dumps(message)) # write to file\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(gen_json)(fil) for fil in files]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "3ab89f3a",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start parallel processing\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "dask.compute(tasks)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cf5adc55",
+ "metadata": {},
+ "source": [
+ "### Combine `Kerchunk` reference `.json` files\n",
+ "\n",
+ "We know that four coordinates are identical for every one of our component datasets - they are not functions of valid_time."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "c6422fad",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create a list of reference json files\n",
+ "output_files = glob.glob(f\"{temp_dir}/*.json\")\n",
+ "\n",
+ "# Combine individual references into single consolidated reference\n",
+ "mzz = MultiZarrToZarr(\n",
+ " output_files,\n",
+ " concat_dims=[\"valid_time\"],\n",
+ " identical_dims=[\"latitude\", \"longitude\", \"heightAboveGround\", \"step\"],\n",
+ ")\n",
+ "multi_kerchunk = mzz.translate()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "59c4209d",
+ "metadata": {},
+ "source": [
+ "### Write combined `Kerchunk` reference file to `.json`"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "b5f9325d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Write Kerchunk .json record\n",
+ "output_fname = \"HRRR_combined.json\"\n",
+ "with open(f\"{output_fname}\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(multi_kerchunk).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "cf74baf0",
+ "metadata": {},
+ "source": [
+ "## Shut down the Dask cluster"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "70e67132",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client.shutdown()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.11"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "b8afa8ad8f3d27e858f1dbdc03ccd45fac432e2a03d4a98c501e197170438b83"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/_preview/58/_sources/notebooks/generating_references/GeoTIFF.ipynb b/_preview/58/_sources/notebooks/generating_references/GeoTIFF.ipynb
new file mode 100644
index 0000000..05593a9
--- /dev/null
+++ b/_preview/58/_sources/notebooks/generating_references/GeoTIFF.ipynb
@@ -0,0 +1,300 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# GeoTIFF\n",
+ "Generating Kerchunk References from GeoTIFF files"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![\"ARG\"](\"../images/radar.png\")
"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ "\n",
+ "In this tutorial we will cover:\n",
+ "\n",
+ "1. How to generate `Kerchunk` references of GeoTIFFs.\n",
+ "1. Combining `Kerchunk` references into a virtual dataset.\n",
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Kerchunk and Dask](../foundations/kerchunk_dask) | Required | Core |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Required | IO/Visualization |\n",
+ "- **Time to learn**: 30 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## About the Dataset\n",
+ "\n",
+ "The Finish Meterological Institute (FMI) Weather Radar Dataset is a collection of GeoTIFF files containing multiple radar specific variables, such as rainfall intensity, precipitation accumulation (in 1, 12 and 24 hour increments), radar reflectivity, radial velocity, rain classification and the cloud top height. It is available through the [AWS public data portal](https://aws.amazon.com/marketplace/pp/prodview-koodet467asui?sr=0-19&ref_=beagle&applicationId=AWSMPContessa) and is updated frequently. \n",
+ "\n",
+ "\n",
+ "\n",
+ "More details on this dataset can be found [here](https://en.ilmatieteenlaitos.fi/radar-data-on-aws-s3).\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import glob\n",
+ "import logging\n",
+ "from tempfile import TemporaryDirectory\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import numpy as np\n",
+ "import rioxarray\n",
+ "import s3fs\n",
+ "import ujson\n",
+ "from distributed import Client\n",
+ "from kerchunk.combine import MultiZarrToZarr\n",
+ "from kerchunk.tiff import tiff_to_zarr"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Examining a Single GeoTIFF File\n",
+ "\n",
+ "Before we use `Kerchunk` to create indices for multiple files, we can load a single GeoTiff file to examine it. \n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# URL pointing to a single GeoTIFF file\n",
+ "url = \"s3://fmi-opendata-radar-geotiff/2023/07/01/FIN-ACRR-3067-1KM/202307010100_FIN-ACRR1H-3067-1KM.tif\"\n",
+ "\n",
+ "# Initialize a s3 filesystem\n",
+ "fs = s3fs.S3FileSystem(anon=True)\n",
+ "\n",
+ "xds = rioxarray.open_rasterio(fs.open(url))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "xds"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "xds.isel(band=0).where(xds < 2000).plot()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create Input File List\n",
+ "\n",
+ "Here we are using `fsspec's` glob functionality along with the *`*`* wildcard operator and some string slicing to grab a list of GeoTIFF files from a `s3` `fsspec` filesystem. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "files_paths = fs_read.glob(\n",
+ " \"s3://fmi-opendata-radar-geotiff/2023/01/01/FIN-ACRR-3067-1KM/*24H-3067-1KM.tif\"\n",
+ ")\n",
+ "# Here we prepend the prefix 's3://', which points to AWS.\n",
+ "file_pattern = sorted([\"s3://\" + f for f in files_paths])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This dictionary will be passed as kwargs to `fsspec`. For more details,\n",
+ "# check out the `foundations/kerchunk_basics` notebook.\n",
+ "so = dict(mode=\"rb\", anon=True, default_fill_cache=False, default_cache_type=\"first\")\n",
+ "\n",
+ "# We are creating a temporary directory to store the .json reference files\n",
+ "# Alternately, you could write these to cloud storage.\n",
+ "td = TemporaryDirectory()\n",
+ "temp_dir = td.name\n",
+ "temp_dir"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Start a Dask Client\n",
+ "\n",
+ "To parallelize the creation of our reference files, we will use `Dask`. For a detailed guide on how to use Dask and Kerchunk, see the Foundations notebook: [Kerchunk and Dask](../foundations/kerchunk_dask).\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use Kerchunk's `tiff_to_zarr` method to create create reference files\n",
+ "\n",
+ "\n",
+ "def generate_json_reference(fil, output_dir: str):\n",
+ " tiff_chunks = tiff_to_zarr(fil, remote_options={\"protocol\": \"s3\", \"anon\": True})\n",
+ " fname = fil.split(\"/\")[-1].split(\"_\")[0]\n",
+ " outf = f\"{output_dir}/{fname}.json\"\n",
+ " with open(outf, \"wb\") as f:\n",
+ " f.write(ujson.dumps(tiff_chunks).encode())\n",
+ " return outf\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil, temp_dir) for fil in file_pattern]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start parallel processing\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "dask.compute(tasks)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Combine Reference Files into Multi-File Reference Dataset\n",
+ "\n",
+ "Now we will combine all the reference files generated into a single reference dataset. Since each TIFF file is a single timeslice and the only temporal information is stored in the filepath, we will have to specify the `coo_map` kwarg in `MultiZarrToZarr` to build a dimension from the filepath attributes. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ref_files = sorted(glob.iglob(f\"{temp_dir}/*.json\"))\n",
+ "\n",
+ "\n",
+ "# Custom Kerchunk function from `coo_map` to create dimensions\n",
+ "def fn_to_time(index, fs, var, fn):\n",
+ " import datetime\n",
+ "\n",
+ " subst = fn.split(\"/\")[-1].split(\".json\")[0]\n",
+ " return datetime.datetime.strptime(subst, \"%Y%m%d%H%M\")\n",
+ "\n",
+ "\n",
+ "mzz = MultiZarrToZarr(\n",
+ " path=ref_files,\n",
+ " indicts=ref_files,\n",
+ " remote_protocol=\"s3\",\n",
+ " remote_options={\"anon\": True},\n",
+ " coo_map={\"time\": fn_to_time},\n",
+ " coo_dtypes={\"time\": np.dtype(\"M8[s]\")},\n",
+ " concat_dims=[\"time\"],\n",
+ " identical_dims=[\"X\", \"Y\"],\n",
+ ")\n",
+ "\n",
+ "# # save translate reference in memory for later visualization\n",
+ "multi_kerchunk = mzz.translate()\n",
+ "\n",
+ "# Write kerchunk .json record\n",
+ "output_fname = \"RADAR.json\"\n",
+ "with open(f\"{output_fname}\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(multi_kerchunk).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Shut down the Dask cluster"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client.shutdown()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.11"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/_preview/58/_sources/notebooks/generating_references/NetCDF.ipynb b/_preview/58/_sources/notebooks/generating_references/NetCDF.ipynb
new file mode 100644
index 0000000..1ceef29
--- /dev/null
+++ b/_preview/58/_sources/notebooks/generating_references/NetCDF.ipynb
@@ -0,0 +1,344 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# NetCDF\n",
+ "Generating Kerchunk References from NetCDF files\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "![\"ARG\"](\"../images/ARG.png\")
"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "Within this notebook, we will cover:\n",
+ "\n",
+ "1. How to access remote NetCDF data using `Kerchunk`\n",
+ "1. Combining multiple `Kerchunk` reference files using `MultiZarrToZarr`\n",
+ "\n",
+ "This notebook shares many similarities with the [Multi-File Datasets with Kerchunk](../foundations/kerchunk_multi_file.ipynb). If you are confused on the function of a block of code, please refer there for a more detailed breakdown of what each line is doing.\n",
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Kerchunk and Dask](../foundations/kerchunk_dask) | Required | Core |\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Motivation\n",
+ "\n",
+ "NetCDF4/HDF5 is one of the most universally adopted file formats in earth sciences, with support of much of the community as well as scientific agencies, data centers and university labs. A huge amount of legacy data has been generated in this format. Fortunately, using `Kerchunk`, we can read these datasets as if they were an Analysis-Read Cloud-Optimized (ARCO) format such as `Zarr`."
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## About the Dataset\n",
+ "\n",
+ "For this example, we will look at a weather dataset composed of multiple NetCDF files.The SMN-Arg is a WRF deterministic weather forecasting dataset created by the `Servicio Meteorológico Nacional de Argentina` that covers Argentina as well as many neighboring countries at a 4km spatial resolution. \n",
+ "The model is initialized twice daily at 00 & 12 UTC with hourly forecasts for variables such as temperature, relative humidity, precipitation, wind direction and magnitude etc. for multiple atmospheric levels.\n",
+ "The data is output at hourly intervals with a maximum prediction lead time of 72 hours in NetCDF files.\n",
+ "\n",
+ "\n",
+ "More details on this dataset can be found [here](https://registry.opendata.aws/smn-ar-wrf-dataset/).\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Flags\n",
+ "In the section below, set the `subset` flag to be `True` (default) or `False` depending if you want this notebook to process the full file list. If set to `True`, then a subset of the file list will be processed (Recommended)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "subset_flag = True"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import glob\n",
+ "import logging\n",
+ "from tempfile import TemporaryDirectory\n",
+ "\n",
+ "import dask\n",
+ "import fsspec\n",
+ "import s3fs\n",
+ "import ujson\n",
+ "import xarray as xr\n",
+ "from distributed import Client\n",
+ "from kerchunk.combine import MultiZarrToZarr\n",
+ "from kerchunk.hdf import SingleHdf5ToZarr"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Examining a Single NetCDF File\n",
+ "\n",
+ "Before we use `Kerchunk` to create indices for multiple files, we can load a single NetCDF file to examine it. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# URL pointing to a single NetCDF file\n",
+ "url = \"s3://smn-ar-wrf/DATA/WRF/DET/2022/12/31/00/WRFDETAR_01H_20221231_00_072.nc\"\n",
+ "\n",
+ "# Initialize a s3 filesystem\n",
+ "fs = s3fs.S3FileSystem(anon=True)\n",
+ "# Use Xarray to open a remote NetCDF file\n",
+ "ds = xr.open_dataset(fs.open(url), engine=\"h5netcdf\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ds"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Here we see the `repr` from the `Xarray` Dataset of a single `NetCDF` file. From examining the output, we can tell that the Dataset dimensions are `['time','y','x']`, with time being only a single step.\n",
+ "Later, when we use `Kerchunk's` `MultiZarrToZarr` functionality, we will need to know on which dimensions to concatenate across. \n",
+ "\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create Input File List\n",
+ "\n",
+ "Here we are using `fsspec's` glob functionality along with the *`*`* wildcard operator and some string slicing to grab a list of NetCDF files from a `s3` `fsspec` filesystem. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Initiate fsspec filesystems for reading\n",
+ "fs_read = fsspec.filesystem(\"s3\", anon=True, skip_instance_cache=True)\n",
+ "\n",
+ "files_paths = fs_read.glob(\"s3://smn-ar-wrf/DATA/WRF/DET/2022/12/31/12/*\")\n",
+ "\n",
+ "# Here we prepend the prefix 's3://', which points to AWS.\n",
+ "file_pattern = sorted([\"s3://\" + f for f in files_paths])\n",
+ "\n",
+ "\n",
+ "# If the subset_flag == True (default), the list of input files will be subset\n",
+ "# to speed up the processing\n",
+ "if subset_flag:\n",
+ " file_pattern = file_pattern[0:8]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# This dictionary will be passed as kwargs to `fsspec`. For more details, check out the\n",
+ "# `foundations/kerchunk_basics` notebook.\n",
+ "so = dict(mode=\"rb\", anon=True, default_fill_cache=False, default_cache_type=\"first\")\n",
+ "\n",
+ "# We are creating a temporary directory to store the .json reference files\n",
+ "# Alternately, you could write these to cloud storage.\n",
+ "td = TemporaryDirectory()\n",
+ "temp_dir = td.name\n",
+ "temp_dir"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Start a Dask Client\n",
+ "\n",
+ "To parallelize the creation of our reference files, we will use `Dask`. For a detailed guide on how to use Dask and Kerchunk, see the Foundations notebook: [Kerchunk and Dask](../foundations/kerchunk_dask).\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8, silence_logs=logging.ERROR)\n",
+ "client"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Use Kerchunk's `SingleHdf5ToZarr` method to create a `Kerchunk` index from\n",
+ "# a NetCDF file.\n",
+ "\n",
+ "\n",
+ "def generate_json_reference(fil, output_dir: str):\n",
+ " with fs_read.open(fil, **so) as infile:\n",
+ " h5chunks = SingleHdf5ToZarr(infile, fil, inline_threshold=300)\n",
+ " fname = fil.split(\"/\")[-1].strip(\".nc\")\n",
+ " outf = f\"{output_dir}/{fname}.json\"\n",
+ " with open(outf, \"wb\") as f:\n",
+ " f.write(ujson.dumps(h5chunks.translate()).encode())\n",
+ " return outf\n",
+ "\n",
+ "\n",
+ "# Generate Dask Delayed objects\n",
+ "tasks = [dask.delayed(generate_json_reference)(fil, temp_dir) for fil in file_pattern]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Start parallel processing\n",
+ "import warnings\n",
+ "\n",
+ "warnings.filterwarnings(\"ignore\")\n",
+ "dask.compute(tasks)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Combine .json `Kerchunk` reference files and write a combined `Kerchunk` index\n",
+ "\n",
+ "In the following cell, we are combining all the `.json` reference files that were generated above into a single reference file and writing that file to disk."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Create a list of reference json files\n",
+ "output_files = glob.glob(f\"{temp_dir}/*.json\")\n",
+ "\n",
+ "# combine individual references into single consolidated reference\n",
+ "mzz = MultiZarrToZarr(\n",
+ " output_files,\n",
+ " concat_dims=[\"time\"],\n",
+ " identical_dims=[\"y\", \"x\"],\n",
+ " remote_options={\"anon\": True},\n",
+ ")\n",
+ "# save translate reference in memory for later visualization\n",
+ "multi_kerchunk = mzz.translate()\n",
+ "\n",
+ "# Write kerchunk .json record.\n",
+ "output_fname = \"ARG_combined.json\"\n",
+ "with open(f\"{output_fname}\", \"wb\") as f:\n",
+ " f.write(ujson.dumps(multi_kerchunk).encode())"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Shut down the Dask cluster"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client.shutdown()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.11"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "89095a95fbc59e1db286735bee0073a08e46abd63daa66f53634eb5c8cc2192a"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/_preview/58/_sources/notebooks/how-to-cite.md b/_preview/58/_sources/notebooks/how-to-cite.md
new file mode 100644
index 0000000..6c38f6b
--- /dev/null
+++ b/_preview/58/_sources/notebooks/how-to-cite.md
@@ -0,0 +1,7 @@
+# How to Cite This Cookbook
+
+The material in Project Pythia's Kerchunk Cookbook is licensed for free and open consumption and reuse. All code is served under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0), while all non-code content is licensed under [Creative Commons BY 4.0 (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/). Effectively, this means you are free to share and adapt this material so long as you give appropriate credit to the Cookbook authors and the Project Pythia community.
+
+The source code for the book is [released on GitHub](https://github.com/ProjectPythia/kerchunk-cookbook) and archived on Zenodo. This DOI will always resolve to the latest release of the book source:
+
+[](https://zenodo.org/badge/latestdoi/588661659)
diff --git a/_preview/58/_sources/notebooks/using_references/Datatree.ipynb b/_preview/58/_sources/notebooks/using_references/Datatree.ipynb
new file mode 100644
index 0000000..c6d6806
--- /dev/null
+++ b/_preview/58/_sources/notebooks/using_references/Datatree.ipynb
@@ -0,0 +1,292 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Kerchunk and Xarray-Datatree"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ "\n",
+ "In this tutorial we are going to use a large collection of pre-generated `Kerchunk` reference files and open them with [xarray-datatree](https://xarray-datatree.readthedocs.io/en/latest/). This chapter is heavily inspired by [this blog post](https://medium.com/pangeo/easy-ipcc-part-1-multi-model-datatree-469b87cf9114).\n",
+ "\n",
+ "\n",
+ "\n",
+ "### About the Dataset\n",
+ "\n",
+ "This collection of reference files were generated from the [**NASA NEX-GDDP-CMIP6 (Global Daily Downscaled Projections)**](https://www.nccs.nasa.gov/services/data-collections/land-based-products/nex-gddp-cmip6) dataset. A version of this dataset is hosted on `s3` as a collection of [NetCDF files](https://registry.opendata.aws/nex-gddp-cmip6/). \n",
+ "\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Multiple Files and Kerchunk](../foundations/kerchunk_multi_file) | Required | Core |\n",
+ "| [Kerchunk and Dask](../foundations/kerchunk_dask) | Required | Core |\n",
+ "| [Multi-File Datasets with Kerchunk](../case_studies/ARG_Weather.ipynb) | Required | IO/Visualization |\n",
+ "| [Xarray-Datatree Overview](https://xarray-datatree.readthedocs.io/en/latest/quick-overview.html)| Required | IO |\n",
+ "\n",
+ "- **Time to learn**: 30 minutes\n",
+ "\n",
+ "## Motivation\n",
+ "\n",
+ "In total the dataset is roughly **12TB** in compressed blob storage, with a single `NetCDF` file per yearly timestep, per variable. Downloading this entire dataset for analysis on a local machine would difficult to say the least. The collection of `Kerchunk` reference files for this entire dataset is only **272 Mb**, which is about 42,000 times smaller! "
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import dask\n",
+ "import hvplot.xarray # noqa\n",
+ "import pandas as pd\n",
+ "import xarray as xr\n",
+ "from datatree import DataTree\n",
+ "from distributed import Client\n",
+ "from fsspec.implementations.reference import ReferenceFileSystem"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Read the reference catalog\n",
+ "\n",
+ "The **NASA NEX-GDDP-CMIP6** dataset is organized by GCM, Scenario and Ensemble Member. Each of these Scenario/GCM combinations is represented as a combined reference file, which was created by merging across variables and concatenating along time-steps. All of these references are organized into a simple `.csv` catalog in the schema: \n",
+ "| GCM/Scenario | url |\n",
+ "| --------------- | ----------- |\n",
+ "\n",
+ "\n",
+ "# Organzing with Xarray-Datatree\n",
+ "Not all of the GCM/Scenario reference datasets have shared spatial coordinates and many of the have slight differences in their calendar and thus time dimension. \n",
+ "Because of this, these cannot be combined into a single `Xarray-Dataset`. Fortunately `Xarray-Datatree` provides a higher level abstraction where related `Xarray-Datasets` are organized into a tree structure where each dataset corresponds to a `leaf`."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Read the reference catalog into a Pandas DataFrame\n",
+ "cat_df = pd.read_csv(\n",
+ " \"s3://carbonplan-share/nasa-nex-reference/reference_catalog_nested.csv\"\n",
+ ")\n",
+ "# Convert the DataFrame into a dictionary\n",
+ "catalog = cat_df.set_index(\"ID\").T.to_dict(\"records\")[0]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Load Reference Datasets into Xarray-DataTree\n",
+ "\n",
+ "In the following cell we create a function `load_ref_ds`, which can be parallelized via `Dask` to load `Kerchunk` references into a dictionary of `Xarray-Datasets`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "def load_ref_ds(url: str):\n",
+ " fs = ReferenceFileSystem(\n",
+ " url,\n",
+ " remote_protocol=\"s3\",\n",
+ " target_protocol=\"s3\",\n",
+ " target_options={\"anon\": True},\n",
+ " lazy=True,\n",
+ " )\n",
+ " return xr.open_dataset(\n",
+ " fs.get_mapper(),\n",
+ " engine=\"zarr\",\n",
+ " backend_kwargs={\"consolidated\": False},\n",
+ " chunks={\"time\": 300},\n",
+ " )\n",
+ "\n",
+ "\n",
+ "tasks = {id: dask.delayed(load_ref_ds)(url) for id, url in catalog.items()}"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Use Dask Distributed to load the Xarray-Datasets from Kerchunk reference files\n",
+ "Using `Dask`, we are loading 164 reference datasets into memory. Since they are are `Xarray` datasets the coordinates are loaded eagerly, but the underlying data is still lazy. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client = Client(n_workers=8)\n",
+ "client"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "catalog_computed = dask.compute(tasks)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Build an Xarray-Datatree from the dictionary of datasets"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dt = DataTree.from_dict(catalog_computed[0])"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Accessing the Datatree\n",
+ "\n",
+ "A `Datatree` is a collection of related `Xarray` datasets. We can access individual datasets using `UNIX` syntax. In the cell below, we will access a single dataset from the `datatree`.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dt[\"ACCESS-CM2/ssp585\"]\n",
+ "\n",
+ "# or\n",
+ "\n",
+ "dt[\"ACCESS-CM2\"][\"ssp585\"]"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "#### Convert a Datatree node to a Dataset"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "dt[\"ACCESS-CM2\"][\"ssp585\"].to_dataset()"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Operations across a Datatree\n",
+ "A `Datatree` contains a collection of datasets with related coordinates and variables. Using some in-built methods, we can analyze it as if it were a single dataset. Instead of looping through hundreds of `Xarray` datasets, we can apply operations across the `Datatree`. In the example below, we will lazily create a time-series."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ts = dt.mean(dim=[\"lat\", \"lon\"])"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Visualize a single dataset with HvPlot"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "display( # noqa\n",
+ " dt[\"ACCESS-CM2/ssp585\"].to_dataset().pr.hvplot(\"lon\", \"lat\", rasterize=True)\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Shut down the Dask cluster"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "client.shutdown()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/_preview/58/_sources/notebooks/using_references/Hvplot_Datashader.ipynb b/_preview/58/_sources/notebooks/using_references/Hvplot_Datashader.ipynb
new file mode 100644
index 0000000..ae81322
--- /dev/null
+++ b/_preview/58/_sources/notebooks/using_references/Hvplot_Datashader.ipynb
@@ -0,0 +1,193 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "f6c39c84",
+ "metadata": {},
+ "source": [
+ "# Kerchunk, hvPlot, and Datashader: Visualizing datasets on-the-fly"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "9eced552",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "This notebook will demonstrate how to use Kerchunk with hvPlot and Datashader to lazily visualize a reference dataset in a streaming fashion.\n",
+ "\n",
+ "We will be building off the references generated through the notebook content from the[Pangeo_Forge](notebooks/generating_references/Pangeo_Forge.ipynb) notebook, so it's encouraged you first go through that.\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Introduction to Xarray](https://foundations.projectpythia.org/core/xarray/xarray-intro.html) | Required | IO |\n",
+ "| [Introduction to hvPlot](https://hvplot.holoviz.org/) | Required | Data Visualization |\n",
+ "| [Introduction to Datashader](https://datashader.org/index.html) | Required | Big Data Visualization |\n",
+ "- **Time to learn**: 10 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "4043365b",
+ "metadata": {},
+ "source": [
+ "## Motivation\n",
+ "\n",
+ "Using Kerchunk, we don't have to create a copy of the data--instead we create a collection of reference files, so that the original data files can be read as if they were Zarr.\n",
+ "\n",
+ "This enables visualization on-the-fly; simply pass in the URL to the dataset and use hvplot."
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "id": "8e2c4765",
+ "metadata": {},
+ "source": [
+ "### Getting to Know The Data\n",
+ "\n",
+ "`gridMET` is a high-resolution daily meteorological dataset covering CONUS from 1979-2023. It is produced by the Climatology Lab at UC Merced. In this example, we are going to look create a virtual Zarr dataset of a derived variable, Burn Index. "
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f8d0f8f1",
+ "metadata": {},
+ "source": [
+ "## Imports"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "706c1b95",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import hvplot.xarray # noqa\n",
+ "import xarray as xr"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "7818440e",
+ "metadata": {},
+ "source": [
+ "## Opening the Kerchunk Dataset\n",
+ "\n",
+ "Now, it's a matter of opening the Kerchunk dataset and calling `hvplot` with the `rasterize=True` keyword argument.\n",
+ "\n",
+ "If you're running this notebook locally, try zooming around the map by hovering over the plot and scrolling; it should update fairly quickly. Note, it will **not** update if you're viewing this on the docs page online as there is no backend server, but don't fret because there's a demo GIF below!"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "ce0f1766",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%timeit -r 1 -n 1\n",
+ "\n",
+ "\n",
+ "storage_options = {\n",
+ " \"remote_protocol\": \"http\",\n",
+ " \"skip_instance_cache\": True,\n",
+ "} # options passed to fsspec\n",
+ "open_dataset_options = {\"chunks\": {}, \"decode_coords\": \"all\"} # opens passed to xarray\n",
+ "\n",
+ "ds_kerchunk = xr.open_dataset(\n",
+ " \"references/Pangeo_Forge/reference.json\",\n",
+ " engine=\"kerchunk\",\n",
+ " storage_options=storage_options,\n",
+ " **open_dataset_options,\n",
+ ")\n",
+ "\n",
+ "display(ds_kerchunk.hvplot(\"lon\", \"lat\", rasterize=True)) # noqa"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "e53531df",
+ "metadata": {},
+ "source": [
+ "![\"Kerchunk](\"../images/kerchunk.gif\")
"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f2c56945",
+ "metadata": {},
+ "source": [
+ "## Comparing Against THREDDS\n",
+ "\n",
+ "Now, we will be repeating the previous cell, but with THREDDS.\n",
+ "\n",
+ "Note how the initial load is longer.\n",
+ "\n",
+ "If you're running the notebook locally (or a demo GIF below), zooming in/out also takes longer to finish buffering as well."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "368ac51d",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "%%timeit -r 1 -n 1\n",
+ "\n",
+ "\n",
+ "def url_gen(year):\n",
+ " return (\n",
+ " f\"http://thredds.northwestknowledge.net:8080/thredds/dodsC/MET/bi/bi_{year}.nc\"\n",
+ " )\n",
+ "\n",
+ "\n",
+ "years = list(range(1979, 1980))\n",
+ "urls_list = [url_gen(year) for year in years]\n",
+ "netcdf_ds = xr.open_mfdataset(urls_list, engine=\"netcdf4\")\n",
+ "display(netcdf_ds.hvplot(\"lon\", \"lat\", rasterize=True)) # noqa"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "6c0dd0cd",
+ "metadata": {},
+ "source": [
+ "![\"THREDDS](\"../images/thredds.gif\")
"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.12"
+ },
+ "vscode": {
+ "interpreter": {
+ "hash": "b8afa8ad8f3d27e858f1dbdc03ccd45fac432e2a03d4a98c501e197170438b83"
+ }
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/_preview/58/_sources/notebooks/using_references/Xarray.ipynb b/_preview/58/_sources/notebooks/using_references/Xarray.ipynb
new file mode 100644
index 0000000..62181f8
--- /dev/null
+++ b/_preview/58/_sources/notebooks/using_references/Xarray.ipynb
@@ -0,0 +1,121 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Load Kerchunked dataset with Xarray\n",
+ "\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "Within this notebook, we will cover:\n",
+ "\n",
+ "1. How to load a Kerchunk pre-generated reference file into Xarray as if it were a Zarr store.\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Xarray Tutorial](https://tutorial.xarray.dev/intro.html) | Required | Core |\n",
+ "\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Opening Reference Dataset with Fsspec and Xarray\n",
+ "One way of using our reference dataset is opening it with `Xarray`. To do this, we will create an `fsspec` filesystem and pass it to `Xarray`. "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# create an fsspec reference filesystem from the Kerchunk output\n",
+ "import fsspec\n",
+ "import xarray as xr\n",
+ "\n",
+ "fs = fsspec.filesystem(\n",
+ " \"reference\",\n",
+ " fo=\"references/ARG_combined.json\",\n",
+ " remote_protocol=\"s3\",\n",
+ " skip_instance_cache=True,\n",
+ ")\n",
+ "m = fs.get_mapper(\"\")\n",
+ "ds = xr.open_dataset(m, engine=\"zarr\", backend_kwargs={\"consolidated\": False})"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Opening Reference Dataset with Xarray and the `Kerchunk` Engine\n",
+ "As of writing, the latest version of Kerchunk supports opening an reference dataset with Xarray without specifically creating an fsspec filesystem. This is the same behavior as the example above, just a few less lines of code. \n",
+ "\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "storage_options = {\n",
+ " \"remote_protocol\": \"s3\",\n",
+ " \"skip_instance_cache\": True,\n",
+ "} # options passed to fsspec\n",
+ "open_dataset_options = {\"chunks\": {}} # opens passed to xarray\n",
+ "\n",
+ "ds = xr.open_dataset(\n",
+ " \"references/ARG_combined.json\",\n",
+ " engine=\"kerchunk\",\n",
+ " storage_options=storage_options,\n",
+ " open_dataset_options=open_dataset_options,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/_preview/58/_sources/notebooks/using_references/Xrefcoord.ipynb b/_preview/58/_sources/notebooks/using_references/Xrefcoord.ipynb
new file mode 100644
index 0000000..4d41f20
--- /dev/null
+++ b/_preview/58/_sources/notebooks/using_references/Xrefcoord.ipynb
@@ -0,0 +1,132 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Use `xrefcoord` to Generate Coordinates\n",
+ "\n",
+ "When using `Kerchunk` to generate reference datasets for GeoTIFF's, only the dimensions are preserved. `xrefcoord` is a small utility that allows us to generate coordinates for these reference datasets using the geospatial metadata. Similar to other accessor add-on libraries for `Xarray` such as `rioxarray` and `xwrf`, `xrefcord` provides an accessor for an `Xarray` dataset. Importing `xrefcoord` allows us to use the `.xref` accessor to access additional methods. \n",
+ "\n",
+ "In this tutorial we will use the `generate_coords` method to build coordinates for the `Xarray` dataset. `xrefcoord` is *very experimental* and makes assumptions about the underlying data, such as each variable shares the same dimensions etc. Use with caution!\n"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Overview\n",
+ " \n",
+ "Within this notebook, we will cover:\n",
+ "\n",
+ "1. How to load a Kerchunk reference dataset created from a collection of GeoTIFFs\n",
+ "1. How to use `xrefcoord` to generate coordinates from a GeoTIFF reference dataset\n",
+ "\n",
+ "## Prerequisites\n",
+ "| Concepts | Importance | Notes |\n",
+ "| --- | --- | --- |\n",
+ "| [Kerchunk Basics](../foundations/kerchunk_basics) | Required | Core |\n",
+ "| [Xarray Tutorial](https://tutorial.xarray.dev/intro.html) | Required | Core |\n",
+ "\n",
+ "- **Time to learn**: 45 minutes\n",
+ "---"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import xarray as xr\n",
+ "import xrefcoord # noqa\n",
+ "\n",
+ "storage_options = {\n",
+ " \"remote_protocol\": \"s3\",\n",
+ " \"skip_instance_cache\": True,\n",
+ "} # options passed to fsspec\n",
+ "open_dataset_options = {\"chunks\": {}} # opens passed to xarray\n",
+ "\n",
+ "ds = xr.open_dataset(\n",
+ " \"references/RADAR.json\",\n",
+ " engine=\"kerchunk\",\n",
+ " storage_options=storage_options,\n",
+ " open_dataset_options=open_dataset_options,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Generate coordinates from reference dataset\n",
+ "ref_ds = ds.xref.generate_coords(time_dim_name=\"time\", x_dim_name=\"X\", y_dim_name=\"Y\")\n",
+ "# Rename to rain accumulation in 24 hour period\n",
+ "ref_ds = ref_ds.rename({\"0\": \"rr24h\"})"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create a Map\n",
+ "\n",
+ "Here we are using `Xarray` to select a single time slice and create a map of 24 hour accumulated rainfall."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ref_ds[\"rr24h\"].where(ref_ds.rr24h < 60000).isel(time=0).plot(robust=True)"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create a Time-Series\n",
+ "\n",
+ "Next we are plotting accumulated rain as a function of time for a specific point."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "ref_ds[\"rr24h\"][:, 700, 700].plot()"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.10.13"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
diff --git a/_preview/58/_sphinx_design_static/design-style.1e8bd061cd6da7fc9cf755528e8ffc24.min.css b/_preview/58/_sphinx_design_static/design-style.1e8bd061cd6da7fc9cf755528e8ffc24.min.css
new file mode 100644
index 0000000..eb19f69
--- /dev/null
+++ b/_preview/58/_sphinx_design_static/design-style.1e8bd061cd6da7fc9cf755528e8ffc24.min.css
@@ -0,0 +1 @@
+.sd-bg-primary{background-color:var(--sd-color-primary) !important}.sd-bg-text-primary{color:var(--sd-color-primary-text) !important}button.sd-bg-primary:focus,button.sd-bg-primary:hover{background-color:var(--sd-color-primary-highlight) !important}a.sd-bg-primary:focus,a.sd-bg-primary:hover{background-color:var(--sd-color-primary-highlight) !important}.sd-bg-secondary{background-color:var(--sd-color-secondary) !important}.sd-bg-text-secondary{color:var(--sd-color-secondary-text) !important}button.sd-bg-secondary:focus,button.sd-bg-secondary:hover{background-color:var(--sd-color-secondary-highlight) !important}a.sd-bg-secondary:focus,a.sd-bg-secondary:hover{background-color:var(--sd-color-secondary-highlight) !important}.sd-bg-success{background-color:var(--sd-color-success) !important}.sd-bg-text-success{color:var(--sd-color-success-text) !important}button.sd-bg-success:focus,button.sd-bg-success:hover{background-color:var(--sd-color-success-highlight) !important}a.sd-bg-success:focus,a.sd-bg-success:hover{background-color:var(--sd-color-success-highlight) !important}.sd-bg-info{background-color:var(--sd-color-info) !important}.sd-bg-text-info{color:var(--sd-color-info-text) !important}button.sd-bg-info:focus,button.sd-bg-info:hover{background-color:var(--sd-color-info-highlight) !important}a.sd-bg-info:focus,a.sd-bg-info:hover{background-color:var(--sd-color-info-highlight) !important}.sd-bg-warning{background-color:var(--sd-color-warning) !important}.sd-bg-text-warning{color:var(--sd-color-warning-text) !important}button.sd-bg-warning:focus,button.sd-bg-warning:hover{background-color:var(--sd-color-warning-highlight) !important}a.sd-bg-warning:focus,a.sd-bg-warning:hover{background-color:var(--sd-color-warning-highlight) !important}.sd-bg-danger{background-color:var(--sd-color-danger) !important}.sd-bg-text-danger{color:var(--sd-color-danger-text) !important}button.sd-bg-danger:focus,button.sd-bg-danger:hover{background-color:var(--sd-color-danger-highlight) !important}a.sd-bg-danger:focus,a.sd-bg-danger:hover{background-color:var(--sd-color-danger-highlight) !important}.sd-bg-light{background-color:var(--sd-color-light) !important}.sd-bg-text-light{color:var(--sd-color-light-text) !important}button.sd-bg-light:focus,button.sd-bg-light:hover{background-color:var(--sd-color-light-highlight) !important}a.sd-bg-light:focus,a.sd-bg-light:hover{background-color:var(--sd-color-light-highlight) !important}.sd-bg-muted{background-color:var(--sd-color-muted) !important}.sd-bg-text-muted{color:var(--sd-color-muted-text) !important}button.sd-bg-muted:focus,button.sd-bg-muted:hover{background-color:var(--sd-color-muted-highlight) !important}a.sd-bg-muted:focus,a.sd-bg-muted:hover{background-color:var(--sd-color-muted-highlight) !important}.sd-bg-dark{background-color:var(--sd-color-dark) !important}.sd-bg-text-dark{color:var(--sd-color-dark-text) !important}button.sd-bg-dark:focus,button.sd-bg-dark:hover{background-color:var(--sd-color-dark-highlight) !important}a.sd-bg-dark:focus,a.sd-bg-dark:hover{background-color:var(--sd-color-dark-highlight) !important}.sd-bg-black{background-color:var(--sd-color-black) !important}.sd-bg-text-black{color:var(--sd-color-black-text) !important}button.sd-bg-black:focus,button.sd-bg-black:hover{background-color:var(--sd-color-black-highlight) !important}a.sd-bg-black:focus,a.sd-bg-black:hover{background-color:var(--sd-color-black-highlight) !important}.sd-bg-white{background-color:var(--sd-color-white) !important}.sd-bg-text-white{color:var(--sd-color-white-text) !important}button.sd-bg-white:focus,button.sd-bg-white:hover{background-color:var(--sd-color-white-highlight) !important}a.sd-bg-white:focus,a.sd-bg-white:hover{background-color:var(--sd-color-white-highlight) !important}.sd-text-primary,.sd-text-primary>p{color:var(--sd-color-primary) !important}a.sd-text-primary:focus,a.sd-text-primary:hover{color:var(--sd-color-primary-highlight) !important}.sd-text-secondary,.sd-text-secondary>p{color:var(--sd-color-secondary) !important}a.sd-text-secondary:focus,a.sd-text-secondary:hover{color:var(--sd-color-secondary-highlight) !important}.sd-text-success,.sd-text-success>p{color:var(--sd-color-success) !important}a.sd-text-success:focus,a.sd-text-success:hover{color:var(--sd-color-success-highlight) !important}.sd-text-info,.sd-text-info>p{color:var(--sd-color-info) !important}a.sd-text-info:focus,a.sd-text-info:hover{color:var(--sd-color-info-highlight) !important}.sd-text-warning,.sd-text-warning>p{color:var(--sd-color-warning) !important}a.sd-text-warning:focus,a.sd-text-warning:hover{color:var(--sd-color-warning-highlight) !important}.sd-text-danger,.sd-text-danger>p{color:var(--sd-color-danger) !important}a.sd-text-danger:focus,a.sd-text-danger:hover{color:var(--sd-color-danger-highlight) !important}.sd-text-light,.sd-text-light>p{color:var(--sd-color-light) !important}a.sd-text-light:focus,a.sd-text-light:hover{color:var(--sd-color-light-highlight) !important}.sd-text-muted,.sd-text-muted>p{color:var(--sd-color-muted) !important}a.sd-text-muted:focus,a.sd-text-muted:hover{color:var(--sd-color-muted-highlight) !important}.sd-text-dark,.sd-text-dark>p{color:var(--sd-color-dark) !important}a.sd-text-dark:focus,a.sd-text-dark:hover{color:var(--sd-color-dark-highlight) !important}.sd-text-black,.sd-text-black>p{color:var(--sd-color-black) !important}a.sd-text-black:focus,a.sd-text-black:hover{color:var(--sd-color-black-highlight) !important}.sd-text-white,.sd-text-white>p{color:var(--sd-color-white) !important}a.sd-text-white:focus,a.sd-text-white:hover{color:var(--sd-color-white-highlight) !important}.sd-outline-primary{border-color:var(--sd-color-primary) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-primary:focus,a.sd-outline-primary:hover{border-color:var(--sd-color-primary-highlight) !important}.sd-outline-secondary{border-color:var(--sd-color-secondary) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-secondary:focus,a.sd-outline-secondary:hover{border-color:var(--sd-color-secondary-highlight) !important}.sd-outline-success{border-color:var(--sd-color-success) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-success:focus,a.sd-outline-success:hover{border-color:var(--sd-color-success-highlight) !important}.sd-outline-info{border-color:var(--sd-color-info) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-info:focus,a.sd-outline-info:hover{border-color:var(--sd-color-info-highlight) !important}.sd-outline-warning{border-color:var(--sd-color-warning) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-warning:focus,a.sd-outline-warning:hover{border-color:var(--sd-color-warning-highlight) !important}.sd-outline-danger{border-color:var(--sd-color-danger) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-danger:focus,a.sd-outline-danger:hover{border-color:var(--sd-color-danger-highlight) !important}.sd-outline-light{border-color:var(--sd-color-light) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-light:focus,a.sd-outline-light:hover{border-color:var(--sd-color-light-highlight) !important}.sd-outline-muted{border-color:var(--sd-color-muted) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-muted:focus,a.sd-outline-muted:hover{border-color:var(--sd-color-muted-highlight) !important}.sd-outline-dark{border-color:var(--sd-color-dark) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-dark:focus,a.sd-outline-dark:hover{border-color:var(--sd-color-dark-highlight) !important}.sd-outline-black{border-color:var(--sd-color-black) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-black:focus,a.sd-outline-black:hover{border-color:var(--sd-color-black-highlight) !important}.sd-outline-white{border-color:var(--sd-color-white) !important;border-style:solid !important;border-width:1px !important}a.sd-outline-white:focus,a.sd-outline-white:hover{border-color:var(--sd-color-white-highlight) !important}.sd-bg-transparent{background-color:transparent !important}.sd-outline-transparent{border-color:transparent !important}.sd-text-transparent{color:transparent !important}.sd-p-0{padding:0 !important}.sd-pt-0,.sd-py-0{padding-top:0 !important}.sd-pr-0,.sd-px-0{padding-right:0 !important}.sd-pb-0,.sd-py-0{padding-bottom:0 !important}.sd-pl-0,.sd-px-0{padding-left:0 !important}.sd-p-1{padding:.25rem !important}.sd-pt-1,.sd-py-1{padding-top:.25rem !important}.sd-pr-1,.sd-px-1{padding-right:.25rem !important}.sd-pb-1,.sd-py-1{padding-bottom:.25rem !important}.sd-pl-1,.sd-px-1{padding-left:.25rem !important}.sd-p-2{padding:.5rem !important}.sd-pt-2,.sd-py-2{padding-top:.5rem !important}.sd-pr-2,.sd-px-2{padding-right:.5rem !important}.sd-pb-2,.sd-py-2{padding-bottom:.5rem !important}.sd-pl-2,.sd-px-2{padding-left:.5rem !important}.sd-p-3{padding:1rem !important}.sd-pt-3,.sd-py-3{padding-top:1rem !important}.sd-pr-3,.sd-px-3{padding-right:1rem !important}.sd-pb-3,.sd-py-3{padding-bottom:1rem !important}.sd-pl-3,.sd-px-3{padding-left:1rem !important}.sd-p-4{padding:1.5rem !important}.sd-pt-4,.sd-py-4{padding-top:1.5rem !important}.sd-pr-4,.sd-px-4{padding-right:1.5rem !important}.sd-pb-4,.sd-py-4{padding-bottom:1.5rem !important}.sd-pl-4,.sd-px-4{padding-left:1.5rem !important}.sd-p-5{padding:3rem !important}.sd-pt-5,.sd-py-5{padding-top:3rem !important}.sd-pr-5,.sd-px-5{padding-right:3rem !important}.sd-pb-5,.sd-py-5{padding-bottom:3rem !important}.sd-pl-5,.sd-px-5{padding-left:3rem !important}.sd-m-auto{margin:auto !important}.sd-mt-auto,.sd-my-auto{margin-top:auto !important}.sd-mr-auto,.sd-mx-auto{margin-right:auto !important}.sd-mb-auto,.sd-my-auto{margin-bottom:auto !important}.sd-ml-auto,.sd-mx-auto{margin-left:auto !important}.sd-m-0{margin:0 !important}.sd-mt-0,.sd-my-0{margin-top:0 !important}.sd-mr-0,.sd-mx-0{margin-right:0 !important}.sd-mb-0,.sd-my-0{margin-bottom:0 !important}.sd-ml-0,.sd-mx-0{margin-left:0 !important}.sd-m-1{margin:.25rem !important}.sd-mt-1,.sd-my-1{margin-top:.25rem !important}.sd-mr-1,.sd-mx-1{margin-right:.25rem !important}.sd-mb-1,.sd-my-1{margin-bottom:.25rem !important}.sd-ml-1,.sd-mx-1{margin-left:.25rem !important}.sd-m-2{margin:.5rem !important}.sd-mt-2,.sd-my-2{margin-top:.5rem !important}.sd-mr-2,.sd-mx-2{margin-right:.5rem !important}.sd-mb-2,.sd-my-2{margin-bottom:.5rem !important}.sd-ml-2,.sd-mx-2{margin-left:.5rem !important}.sd-m-3{margin:1rem !important}.sd-mt-3,.sd-my-3{margin-top:1rem !important}.sd-mr-3,.sd-mx-3{margin-right:1rem !important}.sd-mb-3,.sd-my-3{margin-bottom:1rem !important}.sd-ml-3,.sd-mx-3{margin-left:1rem !important}.sd-m-4{margin:1.5rem !important}.sd-mt-4,.sd-my-4{margin-top:1.5rem !important}.sd-mr-4,.sd-mx-4{margin-right:1.5rem !important}.sd-mb-4,.sd-my-4{margin-bottom:1.5rem !important}.sd-ml-4,.sd-mx-4{margin-left:1.5rem !important}.sd-m-5{margin:3rem !important}.sd-mt-5,.sd-my-5{margin-top:3rem !important}.sd-mr-5,.sd-mx-5{margin-right:3rem !important}.sd-mb-5,.sd-my-5{margin-bottom:3rem !important}.sd-ml-5,.sd-mx-5{margin-left:3rem !important}.sd-w-25{width:25% !important}.sd-w-50{width:50% !important}.sd-w-75{width:75% !important}.sd-w-100{width:100% !important}.sd-w-auto{width:auto !important}.sd-h-25{height:25% !important}.sd-h-50{height:50% !important}.sd-h-75{height:75% !important}.sd-h-100{height:100% !important}.sd-h-auto{height:auto !important}.sd-d-none{display:none !important}.sd-d-inline{display:inline !important}.sd-d-inline-block{display:inline-block !important}.sd-d-block{display:block !important}.sd-d-grid{display:grid !important}.sd-d-flex-row{display:-ms-flexbox !important;display:flex !important;flex-direction:row !important}.sd-d-flex-column{display:-ms-flexbox !important;display:flex !important;flex-direction:column !important}.sd-d-inline-flex{display:-ms-inline-flexbox !important;display:inline-flex !important}@media(min-width: 576px){.sd-d-sm-none{display:none !important}.sd-d-sm-inline{display:inline !important}.sd-d-sm-inline-block{display:inline-block !important}.sd-d-sm-block{display:block !important}.sd-d-sm-grid{display:grid !important}.sd-d-sm-flex{display:-ms-flexbox !important;display:flex !important}.sd-d-sm-inline-flex{display:-ms-inline-flexbox !important;display:inline-flex !important}}@media(min-width: 768px){.sd-d-md-none{display:none !important}.sd-d-md-inline{display:inline !important}.sd-d-md-inline-block{display:inline-block !important}.sd-d-md-block{display:block !important}.sd-d-md-grid{display:grid !important}.sd-d-md-flex{display:-ms-flexbox !important;display:flex !important}.sd-d-md-inline-flex{display:-ms-inline-flexbox !important;display:inline-flex !important}}@media(min-width: 992px){.sd-d-lg-none{display:none !important}.sd-d-lg-inline{display:inline !important}.sd-d-lg-inline-block{display:inline-block !important}.sd-d-lg-block{display:block !important}.sd-d-lg-grid{display:grid !important}.sd-d-lg-flex{display:-ms-flexbox !important;display:flex !important}.sd-d-lg-inline-flex{display:-ms-inline-flexbox !important;display:inline-flex !important}}@media(min-width: 1200px){.sd-d-xl-none{display:none !important}.sd-d-xl-inline{display:inline !important}.sd-d-xl-inline-block{display:inline-block !important}.sd-d-xl-block{display:block !important}.sd-d-xl-grid{display:grid !important}.sd-d-xl-flex{display:-ms-flexbox !important;display:flex !important}.sd-d-xl-inline-flex{display:-ms-inline-flexbox !important;display:inline-flex !important}}.sd-align-major-start{justify-content:flex-start !important}.sd-align-major-end{justify-content:flex-end !important}.sd-align-major-center{justify-content:center !important}.sd-align-major-justify{justify-content:space-between !important}.sd-align-major-spaced{justify-content:space-evenly !important}.sd-align-minor-start{align-items:flex-start !important}.sd-align-minor-end{align-items:flex-end !important}.sd-align-minor-center{align-items:center !important}.sd-align-minor-stretch{align-items:stretch !important}.sd-text-justify{text-align:justify !important}.sd-text-left{text-align:left !important}.sd-text-right{text-align:right !important}.sd-text-center{text-align:center !important}.sd-font-weight-light{font-weight:300 !important}.sd-font-weight-lighter{font-weight:lighter !important}.sd-font-weight-normal{font-weight:400 !important}.sd-font-weight-bold{font-weight:700 !important}.sd-font-weight-bolder{font-weight:bolder !important}.sd-font-italic{font-style:italic !important}.sd-text-decoration-none{text-decoration:none !important}.sd-text-lowercase{text-transform:lowercase !important}.sd-text-uppercase{text-transform:uppercase !important}.sd-text-capitalize{text-transform:capitalize !important}.sd-text-wrap{white-space:normal !important}.sd-text-nowrap{white-space:nowrap !important}.sd-text-truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.sd-fs-1,.sd-fs-1>p{font-size:calc(1.375rem + 1.5vw) !important;line-height:unset !important}.sd-fs-2,.sd-fs-2>p{font-size:calc(1.325rem + 0.9vw) !important;line-height:unset !important}.sd-fs-3,.sd-fs-3>p{font-size:calc(1.3rem + 0.6vw) !important;line-height:unset !important}.sd-fs-4,.sd-fs-4>p{font-size:calc(1.275rem + 0.3vw) !important;line-height:unset !important}.sd-fs-5,.sd-fs-5>p{font-size:1.25rem !important;line-height:unset !important}.sd-fs-6,.sd-fs-6>p{font-size:1rem !important;line-height:unset !important}.sd-border-0{border:0 solid !important}.sd-border-top-0{border-top:0 solid !important}.sd-border-bottom-0{border-bottom:0 solid !important}.sd-border-right-0{border-right:0 solid !important}.sd-border-left-0{border-left:0 solid !important}.sd-border-1{border:1px solid !important}.sd-border-top-1{border-top:1px solid !important}.sd-border-bottom-1{border-bottom:1px solid !important}.sd-border-right-1{border-right:1px solid !important}.sd-border-left-1{border-left:1px solid !important}.sd-border-2{border:2px solid !important}.sd-border-top-2{border-top:2px solid !important}.sd-border-bottom-2{border-bottom:2px solid !important}.sd-border-right-2{border-right:2px solid !important}.sd-border-left-2{border-left:2px solid !important}.sd-border-3{border:3px solid !important}.sd-border-top-3{border-top:3px solid !important}.sd-border-bottom-3{border-bottom:3px solid !important}.sd-border-right-3{border-right:3px solid !important}.sd-border-left-3{border-left:3px solid !important}.sd-border-4{border:4px solid !important}.sd-border-top-4{border-top:4px solid !important}.sd-border-bottom-4{border-bottom:4px solid !important}.sd-border-right-4{border-right:4px solid !important}.sd-border-left-4{border-left:4px solid !important}.sd-border-5{border:5px solid !important}.sd-border-top-5{border-top:5px solid !important}.sd-border-bottom-5{border-bottom:5px solid !important}.sd-border-right-5{border-right:5px solid !important}.sd-border-left-5{border-left:5px solid !important}.sd-rounded-0{border-radius:0 !important}.sd-rounded-1{border-radius:.2rem !important}.sd-rounded-2{border-radius:.3rem !important}.sd-rounded-3{border-radius:.5rem !important}.sd-rounded-pill{border-radius:50rem !important}.sd-rounded-circle{border-radius:50% !important}.shadow-none{box-shadow:none !important}.sd-shadow-sm{box-shadow:0 .125rem .25rem var(--sd-color-shadow) !important}.sd-shadow-md{box-shadow:0 .5rem 1rem var(--sd-color-shadow) !important}.sd-shadow-lg{box-shadow:0 1rem 3rem var(--sd-color-shadow) !important}@keyframes sd-slide-from-left{0%{transform:translateX(-100%)}100%{transform:translateX(0)}}@keyframes sd-slide-from-right{0%{transform:translateX(200%)}100%{transform:translateX(0)}}@keyframes sd-grow100{0%{transform:scale(0);opacity:.5}100%{transform:scale(1);opacity:1}}@keyframes sd-grow50{0%{transform:scale(0.5);opacity:.5}100%{transform:scale(1);opacity:1}}@keyframes sd-grow50-rot20{0%{transform:scale(0.5) rotateZ(-20deg);opacity:.5}75%{transform:scale(1) rotateZ(5deg);opacity:1}95%{transform:scale(1) rotateZ(-1deg);opacity:1}100%{transform:scale(1) rotateZ(0);opacity:1}}.sd-animate-slide-from-left{animation:1s ease-out 0s 1 normal none running sd-slide-from-left}.sd-animate-slide-from-right{animation:1s ease-out 0s 1 normal none running sd-slide-from-right}.sd-animate-grow100{animation:1s ease-out 0s 1 normal none running sd-grow100}.sd-animate-grow50{animation:1s ease-out 0s 1 normal none running sd-grow50}.sd-animate-grow50-rot20{animation:1s ease-out 0s 1 normal none running sd-grow50-rot20}.sd-badge{display:inline-block;padding:.35em .65em;font-size:.75em;font-weight:700;line-height:1;text-align:center;white-space:nowrap;vertical-align:baseline;border-radius:.25rem}.sd-badge:empty{display:none}a.sd-badge{text-decoration:none}.sd-btn .sd-badge{position:relative;top:-1px}.sd-btn{background-color:transparent;border:1px solid transparent;border-radius:.25rem;cursor:pointer;display:inline-block;font-weight:400;font-size:1rem;line-height:1.5;padding:.375rem .75rem;text-align:center;text-decoration:none;transition:color .15s ease-in-out,background-color .15s ease-in-out,border-color .15s ease-in-out,box-shadow .15s ease-in-out;vertical-align:middle;user-select:none;-moz-user-select:none;-ms-user-select:none;-webkit-user-select:none}.sd-btn:hover{text-decoration:none}@media(prefers-reduced-motion: reduce){.sd-btn{transition:none}}.sd-btn-primary,.sd-btn-outline-primary:hover,.sd-btn-outline-primary:focus{color:var(--sd-color-primary-text) !important;background-color:var(--sd-color-primary) !important;border-color:var(--sd-color-primary) !important;border-width:1px !important;border-style:solid !important}.sd-btn-primary:hover,.sd-btn-primary:focus{color:var(--sd-color-primary-text) !important;background-color:var(--sd-color-primary-highlight) !important;border-color:var(--sd-color-primary-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-primary{color:var(--sd-color-primary) !important;border-color:var(--sd-color-primary) !important;border-width:1px !important;border-style:solid !important}.sd-btn-secondary,.sd-btn-outline-secondary:hover,.sd-btn-outline-secondary:focus{color:var(--sd-color-secondary-text) !important;background-color:var(--sd-color-secondary) !important;border-color:var(--sd-color-secondary) !important;border-width:1px !important;border-style:solid !important}.sd-btn-secondary:hover,.sd-btn-secondary:focus{color:var(--sd-color-secondary-text) !important;background-color:var(--sd-color-secondary-highlight) !important;border-color:var(--sd-color-secondary-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-secondary{color:var(--sd-color-secondary) !important;border-color:var(--sd-color-secondary) !important;border-width:1px !important;border-style:solid !important}.sd-btn-success,.sd-btn-outline-success:hover,.sd-btn-outline-success:focus{color:var(--sd-color-success-text) !important;background-color:var(--sd-color-success) !important;border-color:var(--sd-color-success) !important;border-width:1px !important;border-style:solid !important}.sd-btn-success:hover,.sd-btn-success:focus{color:var(--sd-color-success-text) !important;background-color:var(--sd-color-success-highlight) !important;border-color:var(--sd-color-success-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-success{color:var(--sd-color-success) !important;border-color:var(--sd-color-success) !important;border-width:1px !important;border-style:solid !important}.sd-btn-info,.sd-btn-outline-info:hover,.sd-btn-outline-info:focus{color:var(--sd-color-info-text) !important;background-color:var(--sd-color-info) !important;border-color:var(--sd-color-info) !important;border-width:1px !important;border-style:solid !important}.sd-btn-info:hover,.sd-btn-info:focus{color:var(--sd-color-info-text) !important;background-color:var(--sd-color-info-highlight) !important;border-color:var(--sd-color-info-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-info{color:var(--sd-color-info) !important;border-color:var(--sd-color-info) !important;border-width:1px !important;border-style:solid !important}.sd-btn-warning,.sd-btn-outline-warning:hover,.sd-btn-outline-warning:focus{color:var(--sd-color-warning-text) !important;background-color:var(--sd-color-warning) !important;border-color:var(--sd-color-warning) !important;border-width:1px !important;border-style:solid !important}.sd-btn-warning:hover,.sd-btn-warning:focus{color:var(--sd-color-warning-text) !important;background-color:var(--sd-color-warning-highlight) !important;border-color:var(--sd-color-warning-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-warning{color:var(--sd-color-warning) !important;border-color:var(--sd-color-warning) !important;border-width:1px !important;border-style:solid !important}.sd-btn-danger,.sd-btn-outline-danger:hover,.sd-btn-outline-danger:focus{color:var(--sd-color-danger-text) !important;background-color:var(--sd-color-danger) !important;border-color:var(--sd-color-danger) !important;border-width:1px !important;border-style:solid !important}.sd-btn-danger:hover,.sd-btn-danger:focus{color:var(--sd-color-danger-text) !important;background-color:var(--sd-color-danger-highlight) !important;border-color:var(--sd-color-danger-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-danger{color:var(--sd-color-danger) !important;border-color:var(--sd-color-danger) !important;border-width:1px !important;border-style:solid !important}.sd-btn-light,.sd-btn-outline-light:hover,.sd-btn-outline-light:focus{color:var(--sd-color-light-text) !important;background-color:var(--sd-color-light) !important;border-color:var(--sd-color-light) !important;border-width:1px !important;border-style:solid !important}.sd-btn-light:hover,.sd-btn-light:focus{color:var(--sd-color-light-text) !important;background-color:var(--sd-color-light-highlight) !important;border-color:var(--sd-color-light-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-light{color:var(--sd-color-light) !important;border-color:var(--sd-color-light) !important;border-width:1px !important;border-style:solid !important}.sd-btn-muted,.sd-btn-outline-muted:hover,.sd-btn-outline-muted:focus{color:var(--sd-color-muted-text) !important;background-color:var(--sd-color-muted) !important;border-color:var(--sd-color-muted) !important;border-width:1px !important;border-style:solid !important}.sd-btn-muted:hover,.sd-btn-muted:focus{color:var(--sd-color-muted-text) !important;background-color:var(--sd-color-muted-highlight) !important;border-color:var(--sd-color-muted-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-muted{color:var(--sd-color-muted) !important;border-color:var(--sd-color-muted) !important;border-width:1px !important;border-style:solid !important}.sd-btn-dark,.sd-btn-outline-dark:hover,.sd-btn-outline-dark:focus{color:var(--sd-color-dark-text) !important;background-color:var(--sd-color-dark) !important;border-color:var(--sd-color-dark) !important;border-width:1px !important;border-style:solid !important}.sd-btn-dark:hover,.sd-btn-dark:focus{color:var(--sd-color-dark-text) !important;background-color:var(--sd-color-dark-highlight) !important;border-color:var(--sd-color-dark-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-dark{color:var(--sd-color-dark) !important;border-color:var(--sd-color-dark) !important;border-width:1px !important;border-style:solid !important}.sd-btn-black,.sd-btn-outline-black:hover,.sd-btn-outline-black:focus{color:var(--sd-color-black-text) !important;background-color:var(--sd-color-black) !important;border-color:var(--sd-color-black) !important;border-width:1px !important;border-style:solid !important}.sd-btn-black:hover,.sd-btn-black:focus{color:var(--sd-color-black-text) !important;background-color:var(--sd-color-black-highlight) !important;border-color:var(--sd-color-black-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-black{color:var(--sd-color-black) !important;border-color:var(--sd-color-black) !important;border-width:1px !important;border-style:solid !important}.sd-btn-white,.sd-btn-outline-white:hover,.sd-btn-outline-white:focus{color:var(--sd-color-white-text) !important;background-color:var(--sd-color-white) !important;border-color:var(--sd-color-white) !important;border-width:1px !important;border-style:solid !important}.sd-btn-white:hover,.sd-btn-white:focus{color:var(--sd-color-white-text) !important;background-color:var(--sd-color-white-highlight) !important;border-color:var(--sd-color-white-highlight) !important;border-width:1px !important;border-style:solid !important}.sd-btn-outline-white{color:var(--sd-color-white) !important;border-color:var(--sd-color-white) !important;border-width:1px !important;border-style:solid !important}.sd-stretched-link::after{position:absolute;top:0;right:0;bottom:0;left:0;z-index:1;content:""}.sd-hide-link-text{font-size:0}.sd-octicon,.sd-material-icon{display:inline-block;fill:currentColor;vertical-align:middle}.sd-avatar-xs{border-radius:50%;object-fit:cover;object-position:center;width:1rem;height:1rem}.sd-avatar-sm{border-radius:50%;object-fit:cover;object-position:center;width:3rem;height:3rem}.sd-avatar-md{border-radius:50%;object-fit:cover;object-position:center;width:5rem;height:5rem}.sd-avatar-lg{border-radius:50%;object-fit:cover;object-position:center;width:7rem;height:7rem}.sd-avatar-xl{border-radius:50%;object-fit:cover;object-position:center;width:10rem;height:10rem}.sd-avatar-inherit{border-radius:50%;object-fit:cover;object-position:center;width:inherit;height:inherit}.sd-avatar-initial{border-radius:50%;object-fit:cover;object-position:center;width:initial;height:initial}.sd-card{background-clip:border-box;background-color:var(--sd-color-card-background);border:1px solid var(--sd-color-card-border);border-radius:.25rem;color:var(--sd-color-card-text);display:-ms-flexbox;display:flex;-ms-flex-direction:column;flex-direction:column;min-width:0;position:relative;word-wrap:break-word}.sd-card>hr{margin-left:0;margin-right:0}.sd-card-hover:hover{border-color:var(--sd-color-card-border-hover);transform:scale(1.01)}.sd-card-body{-ms-flex:1 1 auto;flex:1 1 auto;padding:1rem 1rem}.sd-card-title{margin-bottom:.5rem}.sd-card-subtitle{margin-top:-0.25rem;margin-bottom:0}.sd-card-text:last-child{margin-bottom:0}.sd-card-link:hover{text-decoration:none}.sd-card-link+.card-link{margin-left:1rem}.sd-card-header{padding:.5rem 1rem;margin-bottom:0;background-color:var(--sd-color-card-header);border-bottom:1px solid var(--sd-color-card-border)}.sd-card-header:first-child{border-radius:calc(0.25rem - 1px) calc(0.25rem - 1px) 0 0}.sd-card-footer{padding:.5rem 1rem;background-color:var(--sd-color-card-footer);border-top:1px solid var(--sd-color-card-border)}.sd-card-footer:last-child{border-radius:0 0 calc(0.25rem - 1px) calc(0.25rem - 1px)}.sd-card-header-tabs{margin-right:-0.5rem;margin-bottom:-0.5rem;margin-left:-0.5rem;border-bottom:0}.sd-card-header-pills{margin-right:-0.5rem;margin-left:-0.5rem}.sd-card-img-overlay{position:absolute;top:0;right:0;bottom:0;left:0;padding:1rem;border-radius:calc(0.25rem - 1px)}.sd-card-img,.sd-card-img-bottom,.sd-card-img-top{width:100%}.sd-card-img,.sd-card-img-top{border-top-left-radius:calc(0.25rem - 1px);border-top-right-radius:calc(0.25rem - 1px)}.sd-card-img,.sd-card-img-bottom{border-bottom-left-radius:calc(0.25rem - 1px);border-bottom-right-radius:calc(0.25rem - 1px)}.sd-cards-carousel{width:100%;display:flex;flex-wrap:nowrap;-ms-flex-direction:row;flex-direction:row;overflow-x:hidden;scroll-snap-type:x mandatory}.sd-cards-carousel.sd-show-scrollbar{overflow-x:auto}.sd-cards-carousel:hover,.sd-cards-carousel:focus{overflow-x:auto}.sd-cards-carousel>.sd-card{flex-shrink:0;scroll-snap-align:start}.sd-cards-carousel>.sd-card:not(:last-child){margin-right:3px}.sd-card-cols-1>.sd-card{width:90%}.sd-card-cols-2>.sd-card{width:45%}.sd-card-cols-3>.sd-card{width:30%}.sd-card-cols-4>.sd-card{width:22.5%}.sd-card-cols-5>.sd-card{width:18%}.sd-card-cols-6>.sd-card{width:15%}.sd-card-cols-7>.sd-card{width:12.8571428571%}.sd-card-cols-8>.sd-card{width:11.25%}.sd-card-cols-9>.sd-card{width:10%}.sd-card-cols-10>.sd-card{width:9%}.sd-card-cols-11>.sd-card{width:8.1818181818%}.sd-card-cols-12>.sd-card{width:7.5%}.sd-container,.sd-container-fluid,.sd-container-lg,.sd-container-md,.sd-container-sm,.sd-container-xl{margin-left:auto;margin-right:auto;padding-left:var(--sd-gutter-x, 0.75rem);padding-right:var(--sd-gutter-x, 0.75rem);width:100%}@media(min-width: 576px){.sd-container-sm,.sd-container{max-width:540px}}@media(min-width: 768px){.sd-container-md,.sd-container-sm,.sd-container{max-width:720px}}@media(min-width: 992px){.sd-container-lg,.sd-container-md,.sd-container-sm,.sd-container{max-width:960px}}@media(min-width: 1200px){.sd-container-xl,.sd-container-lg,.sd-container-md,.sd-container-sm,.sd-container{max-width:1140px}}.sd-row{--sd-gutter-x: 1.5rem;--sd-gutter-y: 0;display:-ms-flexbox;display:flex;-ms-flex-wrap:wrap;flex-wrap:wrap;margin-top:calc(var(--sd-gutter-y) * -1);margin-right:calc(var(--sd-gutter-x) * -0.5);margin-left:calc(var(--sd-gutter-x) * -0.5)}.sd-row>*{box-sizing:border-box;flex-shrink:0;width:100%;max-width:100%;padding-right:calc(var(--sd-gutter-x) * 0.5);padding-left:calc(var(--sd-gutter-x) * 0.5);margin-top:var(--sd-gutter-y)}.sd-col{flex:1 0 0%;-ms-flex:1 0 0%}.sd-row-cols-auto>*{flex:0 0 auto;width:auto}.sd-row-cols-1>*{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-row-cols-2>*{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-row-cols-3>*{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-row-cols-4>*{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-row-cols-5>*{flex:0 0 auto;-ms-flex:0 0 auto;width:20%}.sd-row-cols-6>*{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-row-cols-7>*{flex:0 0 auto;-ms-flex:0 0 auto;width:14.2857142857%}.sd-row-cols-8>*{flex:0 0 auto;-ms-flex:0 0 auto;width:12.5%}.sd-row-cols-9>*{flex:0 0 auto;-ms-flex:0 0 auto;width:11.1111111111%}.sd-row-cols-10>*{flex:0 0 auto;-ms-flex:0 0 auto;width:10%}.sd-row-cols-11>*{flex:0 0 auto;-ms-flex:0 0 auto;width:9.0909090909%}.sd-row-cols-12>*{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}@media(min-width: 576px){.sd-col-sm{flex:1 0 0%;-ms-flex:1 0 0%}.sd-row-cols-sm-auto{flex:1 0 auto;-ms-flex:1 0 auto;width:100%}.sd-row-cols-sm-1>*{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-row-cols-sm-2>*{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-row-cols-sm-3>*{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-row-cols-sm-4>*{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-row-cols-sm-5>*{flex:0 0 auto;-ms-flex:0 0 auto;width:20%}.sd-row-cols-sm-6>*{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-row-cols-sm-7>*{flex:0 0 auto;-ms-flex:0 0 auto;width:14.2857142857%}.sd-row-cols-sm-8>*{flex:0 0 auto;-ms-flex:0 0 auto;width:12.5%}.sd-row-cols-sm-9>*{flex:0 0 auto;-ms-flex:0 0 auto;width:11.1111111111%}.sd-row-cols-sm-10>*{flex:0 0 auto;-ms-flex:0 0 auto;width:10%}.sd-row-cols-sm-11>*{flex:0 0 auto;-ms-flex:0 0 auto;width:9.0909090909%}.sd-row-cols-sm-12>*{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}}@media(min-width: 768px){.sd-col-md{flex:1 0 0%;-ms-flex:1 0 0%}.sd-row-cols-md-auto{flex:1 0 auto;-ms-flex:1 0 auto;width:100%}.sd-row-cols-md-1>*{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-row-cols-md-2>*{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-row-cols-md-3>*{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-row-cols-md-4>*{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-row-cols-md-5>*{flex:0 0 auto;-ms-flex:0 0 auto;width:20%}.sd-row-cols-md-6>*{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-row-cols-md-7>*{flex:0 0 auto;-ms-flex:0 0 auto;width:14.2857142857%}.sd-row-cols-md-8>*{flex:0 0 auto;-ms-flex:0 0 auto;width:12.5%}.sd-row-cols-md-9>*{flex:0 0 auto;-ms-flex:0 0 auto;width:11.1111111111%}.sd-row-cols-md-10>*{flex:0 0 auto;-ms-flex:0 0 auto;width:10%}.sd-row-cols-md-11>*{flex:0 0 auto;-ms-flex:0 0 auto;width:9.0909090909%}.sd-row-cols-md-12>*{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}}@media(min-width: 992px){.sd-col-lg{flex:1 0 0%;-ms-flex:1 0 0%}.sd-row-cols-lg-auto{flex:1 0 auto;-ms-flex:1 0 auto;width:100%}.sd-row-cols-lg-1>*{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-row-cols-lg-2>*{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-row-cols-lg-3>*{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-row-cols-lg-4>*{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-row-cols-lg-5>*{flex:0 0 auto;-ms-flex:0 0 auto;width:20%}.sd-row-cols-lg-6>*{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-row-cols-lg-7>*{flex:0 0 auto;-ms-flex:0 0 auto;width:14.2857142857%}.sd-row-cols-lg-8>*{flex:0 0 auto;-ms-flex:0 0 auto;width:12.5%}.sd-row-cols-lg-9>*{flex:0 0 auto;-ms-flex:0 0 auto;width:11.1111111111%}.sd-row-cols-lg-10>*{flex:0 0 auto;-ms-flex:0 0 auto;width:10%}.sd-row-cols-lg-11>*{flex:0 0 auto;-ms-flex:0 0 auto;width:9.0909090909%}.sd-row-cols-lg-12>*{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}}@media(min-width: 1200px){.sd-col-xl{flex:1 0 0%;-ms-flex:1 0 0%}.sd-row-cols-xl-auto{flex:1 0 auto;-ms-flex:1 0 auto;width:100%}.sd-row-cols-xl-1>*{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-row-cols-xl-2>*{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-row-cols-xl-3>*{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-row-cols-xl-4>*{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-row-cols-xl-5>*{flex:0 0 auto;-ms-flex:0 0 auto;width:20%}.sd-row-cols-xl-6>*{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-row-cols-xl-7>*{flex:0 0 auto;-ms-flex:0 0 auto;width:14.2857142857%}.sd-row-cols-xl-8>*{flex:0 0 auto;-ms-flex:0 0 auto;width:12.5%}.sd-row-cols-xl-9>*{flex:0 0 auto;-ms-flex:0 0 auto;width:11.1111111111%}.sd-row-cols-xl-10>*{flex:0 0 auto;-ms-flex:0 0 auto;width:10%}.sd-row-cols-xl-11>*{flex:0 0 auto;-ms-flex:0 0 auto;width:9.0909090909%}.sd-row-cols-xl-12>*{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}}.sd-col-auto{flex:0 0 auto;-ms-flex:0 0 auto;width:auto}.sd-col-1{flex:0 0 auto;-ms-flex:0 0 auto;width:8.3333333333%}.sd-col-2{flex:0 0 auto;-ms-flex:0 0 auto;width:16.6666666667%}.sd-col-3{flex:0 0 auto;-ms-flex:0 0 auto;width:25%}.sd-col-4{flex:0 0 auto;-ms-flex:0 0 auto;width:33.3333333333%}.sd-col-5{flex:0 0 auto;-ms-flex:0 0 auto;width:41.6666666667%}.sd-col-6{flex:0 0 auto;-ms-flex:0 0 auto;width:50%}.sd-col-7{flex:0 0 auto;-ms-flex:0 0 auto;width:58.3333333333%}.sd-col-8{flex:0 0 auto;-ms-flex:0 0 auto;width:66.6666666667%}.sd-col-9{flex:0 0 auto;-ms-flex:0 0 auto;width:75%}.sd-col-10{flex:0 0 auto;-ms-flex:0 0 auto;width:83.3333333333%}.sd-col-11{flex:0 0 auto;-ms-flex:0 0 auto;width:91.6666666667%}.sd-col-12{flex:0 0 auto;-ms-flex:0 0 auto;width:100%}.sd-g-0,.sd-gy-0{--sd-gutter-y: 0}.sd-g-0,.sd-gx-0{--sd-gutter-x: 0}.sd-g-1,.sd-gy-1{--sd-gutter-y: 0.25rem}.sd-g-1,.sd-gx-1{--sd-gutter-x: 0.25rem}.sd-g-2,.sd-gy-2{--sd-gutter-y: 0.5rem}.sd-g-2,.sd-gx-2{--sd-gutter-x: 0.5rem}.sd-g-3,.sd-gy-3{--sd-gutter-y: 1rem}.sd-g-3,.sd-gx-3{--sd-gutter-x: 1rem}.sd-g-4,.sd-gy-4{--sd-gutter-y: 1.5rem}.sd-g-4,.sd-gx-4{--sd-gutter-x: 1.5rem}.sd-g-5,.sd-gy-5{--sd-gutter-y: 3rem}.sd-g-5,.sd-gx-5{--sd-gutter-x: 3rem}@media(min-width: 576px){.sd-col-sm-auto{-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.sd-col-sm-1{-ms-flex:0 0 auto;flex:0 0 auto;width:8.3333333333%}.sd-col-sm-2{-ms-flex:0 0 auto;flex:0 0 auto;width:16.6666666667%}.sd-col-sm-3{-ms-flex:0 0 auto;flex:0 0 auto;width:25%}.sd-col-sm-4{-ms-flex:0 0 auto;flex:0 0 auto;width:33.3333333333%}.sd-col-sm-5{-ms-flex:0 0 auto;flex:0 0 auto;width:41.6666666667%}.sd-col-sm-6{-ms-flex:0 0 auto;flex:0 0 auto;width:50%}.sd-col-sm-7{-ms-flex:0 0 auto;flex:0 0 auto;width:58.3333333333%}.sd-col-sm-8{-ms-flex:0 0 auto;flex:0 0 auto;width:66.6666666667%}.sd-col-sm-9{-ms-flex:0 0 auto;flex:0 0 auto;width:75%}.sd-col-sm-10{-ms-flex:0 0 auto;flex:0 0 auto;width:83.3333333333%}.sd-col-sm-11{-ms-flex:0 0 auto;flex:0 0 auto;width:91.6666666667%}.sd-col-sm-12{-ms-flex:0 0 auto;flex:0 0 auto;width:100%}.sd-g-sm-0,.sd-gy-sm-0{--sd-gutter-y: 0}.sd-g-sm-0,.sd-gx-sm-0{--sd-gutter-x: 0}.sd-g-sm-1,.sd-gy-sm-1{--sd-gutter-y: 0.25rem}.sd-g-sm-1,.sd-gx-sm-1{--sd-gutter-x: 0.25rem}.sd-g-sm-2,.sd-gy-sm-2{--sd-gutter-y: 0.5rem}.sd-g-sm-2,.sd-gx-sm-2{--sd-gutter-x: 0.5rem}.sd-g-sm-3,.sd-gy-sm-3{--sd-gutter-y: 1rem}.sd-g-sm-3,.sd-gx-sm-3{--sd-gutter-x: 1rem}.sd-g-sm-4,.sd-gy-sm-4{--sd-gutter-y: 1.5rem}.sd-g-sm-4,.sd-gx-sm-4{--sd-gutter-x: 1.5rem}.sd-g-sm-5,.sd-gy-sm-5{--sd-gutter-y: 3rem}.sd-g-sm-5,.sd-gx-sm-5{--sd-gutter-x: 3rem}}@media(min-width: 768px){.sd-col-md-auto{-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.sd-col-md-1{-ms-flex:0 0 auto;flex:0 0 auto;width:8.3333333333%}.sd-col-md-2{-ms-flex:0 0 auto;flex:0 0 auto;width:16.6666666667%}.sd-col-md-3{-ms-flex:0 0 auto;flex:0 0 auto;width:25%}.sd-col-md-4{-ms-flex:0 0 auto;flex:0 0 auto;width:33.3333333333%}.sd-col-md-5{-ms-flex:0 0 auto;flex:0 0 auto;width:41.6666666667%}.sd-col-md-6{-ms-flex:0 0 auto;flex:0 0 auto;width:50%}.sd-col-md-7{-ms-flex:0 0 auto;flex:0 0 auto;width:58.3333333333%}.sd-col-md-8{-ms-flex:0 0 auto;flex:0 0 auto;width:66.6666666667%}.sd-col-md-9{-ms-flex:0 0 auto;flex:0 0 auto;width:75%}.sd-col-md-10{-ms-flex:0 0 auto;flex:0 0 auto;width:83.3333333333%}.sd-col-md-11{-ms-flex:0 0 auto;flex:0 0 auto;width:91.6666666667%}.sd-col-md-12{-ms-flex:0 0 auto;flex:0 0 auto;width:100%}.sd-g-md-0,.sd-gy-md-0{--sd-gutter-y: 0}.sd-g-md-0,.sd-gx-md-0{--sd-gutter-x: 0}.sd-g-md-1,.sd-gy-md-1{--sd-gutter-y: 0.25rem}.sd-g-md-1,.sd-gx-md-1{--sd-gutter-x: 0.25rem}.sd-g-md-2,.sd-gy-md-2{--sd-gutter-y: 0.5rem}.sd-g-md-2,.sd-gx-md-2{--sd-gutter-x: 0.5rem}.sd-g-md-3,.sd-gy-md-3{--sd-gutter-y: 1rem}.sd-g-md-3,.sd-gx-md-3{--sd-gutter-x: 1rem}.sd-g-md-4,.sd-gy-md-4{--sd-gutter-y: 1.5rem}.sd-g-md-4,.sd-gx-md-4{--sd-gutter-x: 1.5rem}.sd-g-md-5,.sd-gy-md-5{--sd-gutter-y: 3rem}.sd-g-md-5,.sd-gx-md-5{--sd-gutter-x: 3rem}}@media(min-width: 992px){.sd-col-lg-auto{-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.sd-col-lg-1{-ms-flex:0 0 auto;flex:0 0 auto;width:8.3333333333%}.sd-col-lg-2{-ms-flex:0 0 auto;flex:0 0 auto;width:16.6666666667%}.sd-col-lg-3{-ms-flex:0 0 auto;flex:0 0 auto;width:25%}.sd-col-lg-4{-ms-flex:0 0 auto;flex:0 0 auto;width:33.3333333333%}.sd-col-lg-5{-ms-flex:0 0 auto;flex:0 0 auto;width:41.6666666667%}.sd-col-lg-6{-ms-flex:0 0 auto;flex:0 0 auto;width:50%}.sd-col-lg-7{-ms-flex:0 0 auto;flex:0 0 auto;width:58.3333333333%}.sd-col-lg-8{-ms-flex:0 0 auto;flex:0 0 auto;width:66.6666666667%}.sd-col-lg-9{-ms-flex:0 0 auto;flex:0 0 auto;width:75%}.sd-col-lg-10{-ms-flex:0 0 auto;flex:0 0 auto;width:83.3333333333%}.sd-col-lg-11{-ms-flex:0 0 auto;flex:0 0 auto;width:91.6666666667%}.sd-col-lg-12{-ms-flex:0 0 auto;flex:0 0 auto;width:100%}.sd-g-lg-0,.sd-gy-lg-0{--sd-gutter-y: 0}.sd-g-lg-0,.sd-gx-lg-0{--sd-gutter-x: 0}.sd-g-lg-1,.sd-gy-lg-1{--sd-gutter-y: 0.25rem}.sd-g-lg-1,.sd-gx-lg-1{--sd-gutter-x: 0.25rem}.sd-g-lg-2,.sd-gy-lg-2{--sd-gutter-y: 0.5rem}.sd-g-lg-2,.sd-gx-lg-2{--sd-gutter-x: 0.5rem}.sd-g-lg-3,.sd-gy-lg-3{--sd-gutter-y: 1rem}.sd-g-lg-3,.sd-gx-lg-3{--sd-gutter-x: 1rem}.sd-g-lg-4,.sd-gy-lg-4{--sd-gutter-y: 1.5rem}.sd-g-lg-4,.sd-gx-lg-4{--sd-gutter-x: 1.5rem}.sd-g-lg-5,.sd-gy-lg-5{--sd-gutter-y: 3rem}.sd-g-lg-5,.sd-gx-lg-5{--sd-gutter-x: 3rem}}@media(min-width: 1200px){.sd-col-xl-auto{-ms-flex:0 0 auto;flex:0 0 auto;width:auto}.sd-col-xl-1{-ms-flex:0 0 auto;flex:0 0 auto;width:8.3333333333%}.sd-col-xl-2{-ms-flex:0 0 auto;flex:0 0 auto;width:16.6666666667%}.sd-col-xl-3{-ms-flex:0 0 auto;flex:0 0 auto;width:25%}.sd-col-xl-4{-ms-flex:0 0 auto;flex:0 0 auto;width:33.3333333333%}.sd-col-xl-5{-ms-flex:0 0 auto;flex:0 0 auto;width:41.6666666667%}.sd-col-xl-6{-ms-flex:0 0 auto;flex:0 0 auto;width:50%}.sd-col-xl-7{-ms-flex:0 0 auto;flex:0 0 auto;width:58.3333333333%}.sd-col-xl-8{-ms-flex:0 0 auto;flex:0 0 auto;width:66.6666666667%}.sd-col-xl-9{-ms-flex:0 0 auto;flex:0 0 auto;width:75%}.sd-col-xl-10{-ms-flex:0 0 auto;flex:0 0 auto;width:83.3333333333%}.sd-col-xl-11{-ms-flex:0 0 auto;flex:0 0 auto;width:91.6666666667%}.sd-col-xl-12{-ms-flex:0 0 auto;flex:0 0 auto;width:100%}.sd-g-xl-0,.sd-gy-xl-0{--sd-gutter-y: 0}.sd-g-xl-0,.sd-gx-xl-0{--sd-gutter-x: 0}.sd-g-xl-1,.sd-gy-xl-1{--sd-gutter-y: 0.25rem}.sd-g-xl-1,.sd-gx-xl-1{--sd-gutter-x: 0.25rem}.sd-g-xl-2,.sd-gy-xl-2{--sd-gutter-y: 0.5rem}.sd-g-xl-2,.sd-gx-xl-2{--sd-gutter-x: 0.5rem}.sd-g-xl-3,.sd-gy-xl-3{--sd-gutter-y: 1rem}.sd-g-xl-3,.sd-gx-xl-3{--sd-gutter-x: 1rem}.sd-g-xl-4,.sd-gy-xl-4{--sd-gutter-y: 1.5rem}.sd-g-xl-4,.sd-gx-xl-4{--sd-gutter-x: 1.5rem}.sd-g-xl-5,.sd-gy-xl-5{--sd-gutter-y: 3rem}.sd-g-xl-5,.sd-gx-xl-5{--sd-gutter-x: 3rem}}.sd-flex-row-reverse{flex-direction:row-reverse !important}details.sd-dropdown{position:relative}details.sd-dropdown .sd-summary-title{font-weight:700;padding-right:3em !important;-moz-user-select:none;-ms-user-select:none;-webkit-user-select:none;user-select:none}details.sd-dropdown:hover{cursor:pointer}details.sd-dropdown .sd-summary-content{cursor:default}details.sd-dropdown summary{list-style:none;padding:1em}details.sd-dropdown summary .sd-octicon.no-title{vertical-align:middle}details.sd-dropdown[open] summary .sd-octicon.no-title{visibility:hidden}details.sd-dropdown summary::-webkit-details-marker{display:none}details.sd-dropdown summary:focus{outline:none}details.sd-dropdown .sd-summary-icon{margin-right:.5em}details.sd-dropdown .sd-summary-icon svg{opacity:.8}details.sd-dropdown summary:hover .sd-summary-up svg,details.sd-dropdown summary:hover .sd-summary-down svg{opacity:1;transform:scale(1.1)}details.sd-dropdown .sd-summary-up svg,details.sd-dropdown .sd-summary-down svg{display:block;opacity:.6}details.sd-dropdown .sd-summary-up,details.sd-dropdown .sd-summary-down{pointer-events:none;position:absolute;right:1em;top:1em}details.sd-dropdown[open]>.sd-summary-title .sd-summary-down{visibility:hidden}details.sd-dropdown:not([open])>.sd-summary-title .sd-summary-up{visibility:hidden}details.sd-dropdown:not([open]).sd-card{border:none}details.sd-dropdown:not([open])>.sd-card-header{border:1px solid var(--sd-color-card-border);border-radius:.25rem}details.sd-dropdown.sd-fade-in[open] summary~*{-moz-animation:sd-fade-in .5s ease-in-out;-webkit-animation:sd-fade-in .5s ease-in-out;animation:sd-fade-in .5s ease-in-out}details.sd-dropdown.sd-fade-in-slide-down[open] summary~*{-moz-animation:sd-fade-in .5s ease-in-out,sd-slide-down .5s ease-in-out;-webkit-animation:sd-fade-in .5s ease-in-out,sd-slide-down .5s ease-in-out;animation:sd-fade-in .5s ease-in-out,sd-slide-down .5s ease-in-out}.sd-col>.sd-dropdown{width:100%}.sd-summary-content>.sd-tab-set:first-child{margin-top:0}@keyframes sd-fade-in{0%{opacity:0}100%{opacity:1}}@keyframes sd-slide-down{0%{transform:translate(0, -10px)}100%{transform:translate(0, 0)}}.sd-tab-set{border-radius:.125rem;display:flex;flex-wrap:wrap;margin:1em 0;position:relative}.sd-tab-set>input{opacity:0;position:absolute}.sd-tab-set>input:checked+label{border-color:var(--sd-color-tabs-underline-active);color:var(--sd-color-tabs-label-active)}.sd-tab-set>input:checked+label+.sd-tab-content{display:block}.sd-tab-set>input:not(:checked)+label:hover{color:var(--sd-color-tabs-label-hover);border-color:var(--sd-color-tabs-underline-hover)}.sd-tab-set>input:focus+label{outline-style:auto}.sd-tab-set>input:not(.focus-visible)+label{outline:none;-webkit-tap-highlight-color:transparent}.sd-tab-set>label{border-bottom:.125rem solid transparent;margin-bottom:0;color:var(--sd-color-tabs-label-inactive);border-color:var(--sd-color-tabs-underline-inactive);cursor:pointer;font-size:var(--sd-fontsize-tabs-label);font-weight:700;padding:1em 1.25em .5em;transition:color 250ms;width:auto;z-index:1}html .sd-tab-set>label:hover{color:var(--sd-color-tabs-label-active)}.sd-col>.sd-tab-set{width:100%}.sd-tab-content{box-shadow:0 -0.0625rem var(--sd-color-tabs-overline),0 .0625rem var(--sd-color-tabs-underline);display:none;order:99;padding-bottom:.75rem;padding-top:.75rem;width:100%}.sd-tab-content>:first-child{margin-top:0 !important}.sd-tab-content>:last-child{margin-bottom:0 !important}.sd-tab-content>.sd-tab-set{margin:0}.sd-sphinx-override,.sd-sphinx-override *{-moz-box-sizing:border-box;-webkit-box-sizing:border-box;box-sizing:border-box}.sd-sphinx-override p{margin-top:0}:root{--sd-color-primary: #0071bc;--sd-color-secondary: #6c757d;--sd-color-success: #28a745;--sd-color-info: #17a2b8;--sd-color-warning: #f0b37e;--sd-color-danger: #dc3545;--sd-color-light: #f8f9fa;--sd-color-muted: #6c757d;--sd-color-dark: #212529;--sd-color-black: black;--sd-color-white: white;--sd-color-primary-highlight: #0060a0;--sd-color-secondary-highlight: #5c636a;--sd-color-success-highlight: #228e3b;--sd-color-info-highlight: #148a9c;--sd-color-warning-highlight: #cc986b;--sd-color-danger-highlight: #bb2d3b;--sd-color-light-highlight: #d3d4d5;--sd-color-muted-highlight: #5c636a;--sd-color-dark-highlight: #1c1f23;--sd-color-black-highlight: black;--sd-color-white-highlight: #d9d9d9;--sd-color-primary-text: #fff;--sd-color-secondary-text: #fff;--sd-color-success-text: #fff;--sd-color-info-text: #fff;--sd-color-warning-text: #212529;--sd-color-danger-text: #fff;--sd-color-light-text: #212529;--sd-color-muted-text: #fff;--sd-color-dark-text: #fff;--sd-color-black-text: #fff;--sd-color-white-text: #212529;--sd-color-shadow: rgba(0, 0, 0, 0.15);--sd-color-card-border: rgba(0, 0, 0, 0.125);--sd-color-card-border-hover: hsla(231, 99%, 66%, 1);--sd-color-card-background: transparent;--sd-color-card-text: inherit;--sd-color-card-header: transparent;--sd-color-card-footer: transparent;--sd-color-tabs-label-active: hsla(231, 99%, 66%, 1);--sd-color-tabs-label-hover: hsla(231, 99%, 66%, 1);--sd-color-tabs-label-inactive: hsl(0, 0%, 66%);--sd-color-tabs-underline-active: hsla(231, 99%, 66%, 1);--sd-color-tabs-underline-hover: rgba(178, 206, 245, 0.62);--sd-color-tabs-underline-inactive: transparent;--sd-color-tabs-overline: rgb(222, 222, 222);--sd-color-tabs-underline: rgb(222, 222, 222);--sd-fontsize-tabs-label: 1rem}
diff --git a/_preview/58/_sphinx_design_static/design-tabs.js b/_preview/58/_sphinx_design_static/design-tabs.js
new file mode 100644
index 0000000..36b38cf
--- /dev/null
+++ b/_preview/58/_sphinx_design_static/design-tabs.js
@@ -0,0 +1,27 @@
+var sd_labels_by_text = {};
+
+function ready() {
+ const li = document.getElementsByClassName("sd-tab-label");
+ for (const label of li) {
+ syncId = label.getAttribute("data-sync-id");
+ if (syncId) {
+ label.onclick = onLabelClick;
+ if (!sd_labels_by_text[syncId]) {
+ sd_labels_by_text[syncId] = [];
+ }
+ sd_labels_by_text[syncId].push(label);
+ }
+ }
+}
+
+function onLabelClick() {
+ // Activate other inputs with the same sync id.
+ syncId = this.getAttribute("data-sync-id");
+ for (label of sd_labels_by_text[syncId]) {
+ if (label === this) continue;
+ label.previousElementSibling.checked = true;
+ }
+ window.localStorage.setItem("sphinx-design-last-tab", syncId);
+}
+
+document.addEventListener("DOMContentLoaded", ready, false);
diff --git a/_preview/58/_static/basic.css b/_preview/58/_static/basic.css
new file mode 100644
index 0000000..e760386
--- /dev/null
+++ b/_preview/58/_static/basic.css
@@ -0,0 +1,925 @@
+/*
+ * basic.css
+ * ~~~~~~~~~
+ *
+ * Sphinx stylesheet -- basic theme.
+ *
+ * :copyright: Copyright 2007-2023 by the Sphinx team, see AUTHORS.
+ * :license: BSD, see LICENSE for details.
+ *
+ */
+
+/* -- main layout ----------------------------------------------------------- */
+
+div.clearer {
+ clear: both;
+}
+
+div.section::after {
+ display: block;
+ content: '';
+ clear: left;
+}
+
+/* -- relbar ---------------------------------------------------------------- */
+
+div.related {
+ width: 100%;
+ font-size: 90%;
+}
+
+div.related h3 {
+ display: none;
+}
+
+div.related ul {
+ margin: 0;
+ padding: 0 0 0 10px;
+ list-style: none;
+}
+
+div.related li {
+ display: inline;
+}
+
+div.related li.right {
+ float: right;
+ margin-right: 5px;
+}
+
+/* -- sidebar --------------------------------------------------------------- */
+
+div.sphinxsidebarwrapper {
+ padding: 10px 5px 0 10px;
+}
+
+div.sphinxsidebar {
+ float: left;
+ width: 270px;
+ margin-left: -100%;
+ font-size: 90%;
+ word-wrap: break-word;
+ overflow-wrap : break-word;
+}
+
+div.sphinxsidebar ul {
+ list-style: none;
+}
+
+div.sphinxsidebar ul ul,
+div.sphinxsidebar ul.want-points {
+ margin-left: 20px;
+ list-style: square;
+}
+
+div.sphinxsidebar ul ul {
+ margin-top: 0;
+ margin-bottom: 0;
+}
+
+div.sphinxsidebar form {
+ margin-top: 10px;
+}
+
+div.sphinxsidebar input {
+ border: 1px solid #98dbcc;
+ font-family: sans-serif;
+ font-size: 1em;
+}
+
+div.sphinxsidebar #searchbox form.search {
+ overflow: hidden;
+}
+
+div.sphinxsidebar #searchbox input[type="text"] {
+ float: left;
+ width: 80%;
+ padding: 0.25em;
+ box-sizing: border-box;
+}
+
+div.sphinxsidebar #searchbox input[type="submit"] {
+ float: left;
+ width: 20%;
+ border-left: none;
+ padding: 0.25em;
+ box-sizing: border-box;
+}
+
+
+img {
+ border: 0;
+ max-width: 100%;
+}
+
+/* -- search page ----------------------------------------------------------- */
+
+ul.search {
+ margin: 10px 0 0 20px;
+ padding: 0;
+}
+
+ul.search li {
+ padding: 5px 0 5px 20px;
+ background-image: url(file.png);
+ background-repeat: no-repeat;
+ background-position: 0 7px;
+}
+
+ul.search li a {
+ font-weight: bold;
+}
+
+ul.search li p.context {
+ color: #888;
+ margin: 2px 0 0 30px;
+ text-align: left;
+}
+
+ul.keywordmatches li.goodmatch a {
+ font-weight: bold;
+}
+
+/* -- index page ------------------------------------------------------------ */
+
+table.contentstable {
+ width: 90%;
+ margin-left: auto;
+ margin-right: auto;
+}
+
+table.contentstable p.biglink {
+ line-height: 150%;
+}
+
+a.biglink {
+ font-size: 1.3em;
+}
+
+span.linkdescr {
+ font-style: italic;
+ padding-top: 5px;
+ font-size: 90%;
+}
+
+/* -- general index --------------------------------------------------------- */
+
+table.indextable {
+ width: 100%;
+}
+
+table.indextable td {
+ text-align: left;
+ vertical-align: top;
+}
+
+table.indextable ul {
+ margin-top: 0;
+ margin-bottom: 0;
+ list-style-type: none;
+}
+
+table.indextable > tbody > tr > td > ul {
+ padding-left: 0em;
+}
+
+table.indextable tr.pcap {
+ height: 10px;
+}
+
+table.indextable tr.cap {
+ margin-top: 10px;
+ background-color: #f2f2f2;
+}
+
+img.toggler {
+ margin-right: 3px;
+ margin-top: 3px;
+ cursor: pointer;
+}
+
+div.modindex-jumpbox {
+ border-top: 1px solid #ddd;
+ border-bottom: 1px solid #ddd;
+ margin: 1em 0 1em 0;
+ padding: 0.4em;
+}
+
+div.genindex-jumpbox {
+ border-top: 1px solid #ddd;
+ border-bottom: 1px solid #ddd;
+ margin: 1em 0 1em 0;
+ padding: 0.4em;
+}
+
+/* -- domain module index --------------------------------------------------- */
+
+table.modindextable td {
+ padding: 2px;
+ border-collapse: collapse;
+}
+
+/* -- general body styles --------------------------------------------------- */
+
+div.body {
+ min-width: 360px;
+ max-width: 800px;
+}
+
+div.body p, div.body dd, div.body li, div.body blockquote {
+ -moz-hyphens: auto;
+ -ms-hyphens: auto;
+ -webkit-hyphens: auto;
+ hyphens: auto;
+}
+
+a.headerlink {
+ visibility: hidden;
+}
+
+a:visited {
+ color: #551A8B;
+}
+
+h1:hover > a.headerlink,
+h2:hover > a.headerlink,
+h3:hover > a.headerlink,
+h4:hover > a.headerlink,
+h5:hover > a.headerlink,
+h6:hover > a.headerlink,
+dt:hover > a.headerlink,
+caption:hover > a.headerlink,
+p.caption:hover > a.headerlink,
+div.code-block-caption:hover > a.headerlink {
+ visibility: visible;
+}
+
+div.body p.caption {
+ text-align: inherit;
+}
+
+div.body td {
+ text-align: left;
+}
+
+.first {
+ margin-top: 0 !important;
+}
+
+p.rubric {
+ margin-top: 30px;
+ font-weight: bold;
+}
+
+img.align-left, figure.align-left, .figure.align-left, object.align-left {
+ clear: left;
+ float: left;
+ margin-right: 1em;
+}
+
+img.align-right, figure.align-right, .figure.align-right, object.align-right {
+ clear: right;
+ float: right;
+ margin-left: 1em;
+}
+
+img.align-center, figure.align-center, .figure.align-center, object.align-center {
+ display: block;
+ margin-left: auto;
+ margin-right: auto;
+}
+
+img.align-default, figure.align-default, .figure.align-default {
+ display: block;
+ margin-left: auto;
+ margin-right: auto;
+}
+
+.align-left {
+ text-align: left;
+}
+
+.align-center {
+ text-align: center;
+}
+
+.align-default {
+ text-align: center;
+}
+
+.align-right {
+ text-align: right;
+}
+
+/* -- sidebars -------------------------------------------------------------- */
+
+div.sidebar,
+aside.sidebar {
+ margin: 0 0 0.5em 1em;
+ border: 1px solid #ddb;
+ padding: 7px;
+ background-color: #ffe;
+ width: 40%;
+ float: right;
+ clear: right;
+ overflow-x: auto;
+}
+
+p.sidebar-title {
+ font-weight: bold;
+}
+
+nav.contents,
+aside.topic,
+div.admonition, div.topic, blockquote {
+ clear: left;
+}
+
+/* -- topics ---------------------------------------------------------------- */
+
+nav.contents,
+aside.topic,
+div.topic {
+ border: 1px solid #ccc;
+ padding: 7px;
+ margin: 10px 0 10px 0;
+}
+
+p.topic-title {
+ font-size: 1.1em;
+ font-weight: bold;
+ margin-top: 10px;
+}
+
+/* -- admonitions ----------------------------------------------------------- */
+
+div.admonition {
+ margin-top: 10px;
+ margin-bottom: 10px;
+ padding: 7px;
+}
+
+div.admonition dt {
+ font-weight: bold;
+}
+
+p.admonition-title {
+ margin: 0px 10px 5px 0px;
+ font-weight: bold;
+}
+
+div.body p.centered {
+ text-align: center;
+ margin-top: 25px;
+}
+
+/* -- content of sidebars/topics/admonitions -------------------------------- */
+
+div.sidebar > :last-child,
+aside.sidebar > :last-child,
+nav.contents > :last-child,
+aside.topic > :last-child,
+div.topic > :last-child,
+div.admonition > :last-child {
+ margin-bottom: 0;
+}
+
+div.sidebar::after,
+aside.sidebar::after,
+nav.contents::after,
+aside.topic::after,
+div.topic::after,
+div.admonition::after,
+blockquote::after {
+ display: block;
+ content: '';
+ clear: both;
+}
+
+/* -- tables ---------------------------------------------------------------- */
+
+table.docutils {
+ margin-top: 10px;
+ margin-bottom: 10px;
+ border: 0;
+ border-collapse: collapse;
+}
+
+table.align-center {
+ margin-left: auto;
+ margin-right: auto;
+}
+
+table.align-default {
+ margin-left: auto;
+ margin-right: auto;
+}
+
+table caption span.caption-number {
+ font-style: italic;
+}
+
+table caption span.caption-text {
+}
+
+table.docutils td, table.docutils th {
+ padding: 1px 8px 1px 5px;
+ border-top: 0;
+ border-left: 0;
+ border-right: 0;
+ border-bottom: 1px solid #aaa;
+}
+
+th {
+ text-align: left;
+ padding-right: 5px;
+}
+
+table.citation {
+ border-left: solid 1px gray;
+ margin-left: 1px;
+}
+
+table.citation td {
+ border-bottom: none;
+}
+
+th > :first-child,
+td > :first-child {
+ margin-top: 0px;
+}
+
+th > :last-child,
+td > :last-child {
+ margin-bottom: 0px;
+}
+
+/* -- figures --------------------------------------------------------------- */
+
+div.figure, figure {
+ margin: 0.5em;
+ padding: 0.5em;
+}
+
+div.figure p.caption, figcaption {
+ padding: 0.3em;
+}
+
+div.figure p.caption span.caption-number,
+figcaption span.caption-number {
+ font-style: italic;
+}
+
+div.figure p.caption span.caption-text,
+figcaption span.caption-text {
+}
+
+/* -- field list styles ----------------------------------------------------- */
+
+table.field-list td, table.field-list th {
+ border: 0 !important;
+}
+
+.field-list ul {
+ margin: 0;
+ padding-left: 1em;
+}
+
+.field-list p {
+ margin: 0;
+}
+
+.field-name {
+ -moz-hyphens: manual;
+ -ms-hyphens: manual;
+ -webkit-hyphens: manual;
+ hyphens: manual;
+}
+
+/* -- hlist styles ---------------------------------------------------------- */
+
+table.hlist {
+ margin: 1em 0;
+}
+
+table.hlist td {
+ vertical-align: top;
+}
+
+/* -- object description styles --------------------------------------------- */
+
+.sig {
+ font-family: 'Consolas', 'Menlo', 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', monospace;
+}
+
+.sig-name, code.descname {
+ background-color: transparent;
+ font-weight: bold;
+}
+
+.sig-name {
+ font-size: 1.1em;
+}
+
+code.descname {
+ font-size: 1.2em;
+}
+
+.sig-prename, code.descclassname {
+ background-color: transparent;
+}
+
+.optional {
+ font-size: 1.3em;
+}
+
+.sig-paren {
+ font-size: larger;
+}
+
+.sig-param.n {
+ font-style: italic;
+}
+
+/* C++ specific styling */
+
+.sig-inline.c-texpr,
+.sig-inline.cpp-texpr {
+ font-family: unset;
+}
+
+.sig.c .k, .sig.c .kt,
+.sig.cpp .k, .sig.cpp .kt {
+ color: #0033B3;
+}
+
+.sig.c .m,
+.sig.cpp .m {
+ color: #1750EB;
+}
+
+.sig.c .s, .sig.c .sc,
+.sig.cpp .s, .sig.cpp .sc {
+ color: #067D17;
+}
+
+
+/* -- other body styles ----------------------------------------------------- */
+
+ol.arabic {
+ list-style: decimal;
+}
+
+ol.loweralpha {
+ list-style: lower-alpha;
+}
+
+ol.upperalpha {
+ list-style: upper-alpha;
+}
+
+ol.lowerroman {
+ list-style: lower-roman;
+}
+
+ol.upperroman {
+ list-style: upper-roman;
+}
+
+:not(li) > ol > li:first-child > :first-child,
+:not(li) > ul > li:first-child > :first-child {
+ margin-top: 0px;
+}
+
+:not(li) > ol > li:last-child > :last-child,
+:not(li) > ul > li:last-child > :last-child {
+ margin-bottom: 0px;
+}
+
+ol.simple ol p,
+ol.simple ul p,
+ul.simple ol p,
+ul.simple ul p {
+ margin-top: 0;
+}
+
+ol.simple > li:not(:first-child) > p,
+ul.simple > li:not(:first-child) > p {
+ margin-top: 0;
+}
+
+ol.simple p,
+ul.simple p {
+ margin-bottom: 0;
+}
+
+aside.footnote > span,
+div.citation > span {
+ float: left;
+}
+aside.footnote > span:last-of-type,
+div.citation > span:last-of-type {
+ padding-right: 0.5em;
+}
+aside.footnote > p {
+ margin-left: 2em;
+}
+div.citation > p {
+ margin-left: 4em;
+}
+aside.footnote > p:last-of-type,
+div.citation > p:last-of-type {
+ margin-bottom: 0em;
+}
+aside.footnote > p:last-of-type:after,
+div.citation > p:last-of-type:after {
+ content: "";
+ clear: both;
+}
+
+dl.field-list {
+ display: grid;
+ grid-template-columns: fit-content(30%) auto;
+}
+
+dl.field-list > dt {
+ font-weight: bold;
+ word-break: break-word;
+ padding-left: 0.5em;
+ padding-right: 5px;
+}
+
+dl.field-list > dd {
+ padding-left: 0.5em;
+ margin-top: 0em;
+ margin-left: 0em;
+ margin-bottom: 0em;
+}
+
+dl {
+ margin-bottom: 15px;
+}
+
+dd > :first-child {
+ margin-top: 0px;
+}
+
+dd ul, dd table {
+ margin-bottom: 10px;
+}
+
+dd {
+ margin-top: 3px;
+ margin-bottom: 10px;
+ margin-left: 30px;
+}
+
+.sig dd {
+ margin-top: 0px;
+ margin-bottom: 0px;
+}
+
+.sig dl {
+ margin-top: 0px;
+ margin-bottom: 0px;
+}
+
+dl > dd:last-child,
+dl > dd:last-child > :last-child {
+ margin-bottom: 0;
+}
+
+dt:target, span.highlighted {
+ background-color: #fbe54e;
+}
+
+rect.highlighted {
+ fill: #fbe54e;
+}
+
+dl.glossary dt {
+ font-weight: bold;
+ font-size: 1.1em;
+}
+
+.versionmodified {
+ font-style: italic;
+}
+
+.system-message {
+ background-color: #fda;
+ padding: 5px;
+ border: 3px solid red;
+}
+
+.footnote:target {
+ background-color: #ffa;
+}
+
+.line-block {
+ display: block;
+ margin-top: 1em;
+ margin-bottom: 1em;
+}
+
+.line-block .line-block {
+ margin-top: 0;
+ margin-bottom: 0;
+ margin-left: 1.5em;
+}
+
+.guilabel, .menuselection {
+ font-family: sans-serif;
+}
+
+.accelerator {
+ text-decoration: underline;
+}
+
+.classifier {
+ font-style: oblique;
+}
+
+.classifier:before {
+ font-style: normal;
+ margin: 0 0.5em;
+ content: ":";
+ display: inline-block;
+}
+
+abbr, acronym {
+ border-bottom: dotted 1px;
+ cursor: help;
+}
+
+.translated {
+ background-color: rgba(207, 255, 207, 0.2)
+}
+
+.untranslated {
+ background-color: rgba(255, 207, 207, 0.2)
+}
+
+/* -- code displays --------------------------------------------------------- */
+
+pre {
+ overflow: auto;
+ overflow-y: hidden; /* fixes display issues on Chrome browsers */
+}
+
+pre, div[class*="highlight-"] {
+ clear: both;
+}
+
+span.pre {
+ -moz-hyphens: none;
+ -ms-hyphens: none;
+ -webkit-hyphens: none;
+ hyphens: none;
+ white-space: nowrap;
+}
+
+div[class*="highlight-"] {
+ margin: 1em 0;
+}
+
+td.linenos pre {
+ border: 0;
+ background-color: transparent;
+ color: #aaa;
+}
+
+table.highlighttable {
+ display: block;
+}
+
+table.highlighttable tbody {
+ display: block;
+}
+
+table.highlighttable tr {
+ display: flex;
+}
+
+table.highlighttable td {
+ margin: 0;
+ padding: 0;
+}
+
+table.highlighttable td.linenos {
+ padding-right: 0.5em;
+}
+
+table.highlighttable td.code {
+ flex: 1;
+ overflow: hidden;
+}
+
+.highlight .hll {
+ display: block;
+}
+
+div.highlight pre,
+table.highlighttable pre {
+ margin: 0;
+}
+
+div.code-block-caption + div {
+ margin-top: 0;
+}
+
+div.code-block-caption {
+ margin-top: 1em;
+ padding: 2px 5px;
+ font-size: small;
+}
+
+div.code-block-caption code {
+ background-color: transparent;
+}
+
+table.highlighttable td.linenos,
+span.linenos,
+div.highlight span.gp { /* gp: Generic.Prompt */
+ user-select: none;
+ -webkit-user-select: text; /* Safari fallback only */
+ -webkit-user-select: none; /* Chrome/Safari */
+ -moz-user-select: none; /* Firefox */
+ -ms-user-select: none; /* IE10+ */
+}
+
+div.code-block-caption span.caption-number {
+ padding: 0.1em 0.3em;
+ font-style: italic;
+}
+
+div.code-block-caption span.caption-text {
+}
+
+div.literal-block-wrapper {
+ margin: 1em 0;
+}
+
+code.xref, a code {
+ background-color: transparent;
+ font-weight: bold;
+}
+
+h1 code, h2 code, h3 code, h4 code, h5 code, h6 code {
+ background-color: transparent;
+}
+
+.viewcode-link {
+ float: right;
+}
+
+.viewcode-back {
+ float: right;
+ font-family: sans-serif;
+}
+
+div.viewcode-block:target {
+ margin: -1px -10px;
+ padding: 0 10px;
+}
+
+/* -- math display ---------------------------------------------------------- */
+
+img.math {
+ vertical-align: middle;
+}
+
+div.body div.math p {
+ text-align: center;
+}
+
+span.eqno {
+ float: right;
+}
+
+span.eqno a.headerlink {
+ position: absolute;
+ z-index: 1;
+}
+
+div.math:hover a.headerlink {
+ visibility: visible;
+}
+
+/* -- printout stylesheet --------------------------------------------------- */
+
+@media print {
+ div.document,
+ div.documentwrapper,
+ div.bodywrapper {
+ margin: 0 !important;
+ width: 100%;
+ }
+
+ div.sphinxsidebar,
+ div.related,
+ div.footer,
+ #top-link {
+ display: none;
+ }
+}
\ No newline at end of file
diff --git a/_preview/58/_static/check-solid.svg b/_preview/58/_static/check-solid.svg
new file mode 100644
index 0000000..92fad4b
--- /dev/null
+++ b/_preview/58/_static/check-solid.svg
@@ -0,0 +1,4 @@
+
diff --git a/_preview/58/_static/clipboard.min.js b/_preview/58/_static/clipboard.min.js
new file mode 100644
index 0000000..54b3c46
--- /dev/null
+++ b/_preview/58/_static/clipboard.min.js
@@ -0,0 +1,7 @@
+/*!
+ * clipboard.js v2.0.8
+ * https://clipboardjs.com/
+ *
+ * Licensed MIT © Zeno Rocha
+ */
+!function(t,e){"object"==typeof exports&&"object"==typeof module?module.exports=e():"function"==typeof define&&define.amd?define([],e):"object"==typeof exports?exports.ClipboardJS=e():t.ClipboardJS=e()}(this,function(){return n={686:function(t,e,n){"use strict";n.d(e,{default:function(){return o}});var e=n(279),i=n.n(e),e=n(370),u=n.n(e),e=n(817),c=n.n(e);function a(t){try{return document.execCommand(t)}catch(t){return}}var f=function(t){t=c()(t);return a("cut"),t};var l=function(t){var e,n,o,r=1