data_availability.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "dddb64f8-7c77-4367-81e9-10467c1e7f9a",
   "metadata": {
    "tags": []
   },
   "source": [
    "# EERIE Phase 1 production simulation data available at DKRZ Levante\n",
    "\n",
    "This notebook provides an overview about the EERIE Phase 1 production simulation data available at DKRZ's High Performance Computer Levante. We browse through the [EERIE intake catalog](https://github.com/eerie-project/intake_catalogues/tree/main), search for phase 1 models, experiments and versions and collect information from a datasets:\n",
    "\n",
    "## Phase 1 simulation defintion:\n",
    "\n",
    "- **Source**\n",
    "\n",
    "    The source refers to the EERIE Earth System Model that was used to generate phase 1 production experiments\n",
    "\n",
    "- **Experiments**\n",
    "\n",
    "    For each source, we define a set of phase 1 production experiments for which output is available. Each set contains a keyvalue\n",
    "    - the latest version. Versions in the catalogue distinguish between preliminary and production simulations.\n",
    "    - an example dataset for getting statistical information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a370d9c4-d823-4f98-81a8-f8109c36b233",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#example must be defined\n",
    "#version must be a list\n",
    "phase1_simulations={\n",
    "    \"ifs-fesom2-sr\":[dict(\n",
    "        experiment=\"eerie-spinup-1950\",\n",
    "        version=[\"v20240304\"],\n",
    "        example=\"ocean.native.daily\"\n",
    "    )],\n",
    "    \"icon-esm-er\":[dict(\n",
    "        experiment=\"eerie-spinup-1950\",\n",
    "        version=[\"v20240618\"],\n",
    "        example=\"ocean.native.2d_daily_mean\"\n",
    "    )],\n",
    "    \"ifs-amip\":{},\n",
    "    \"ifs-nemo\":{},\n",
    "    \"hadgem3-gc5-n640-orca12\":[dict(\n",
    "        experiment=\"eerie-picontrol\",\n",
    "        example=\"atmos.native.atmos_monthly_emon\"\n",
    "    )],\n",
    "    \"hadgem3-gc5-n216-orca025\":[dict(\n",
    "        experiment=\"eerie-picontrol\",\n",
    "        example=\"atmos.native.atmos_monthly_emon\"\n",
    "    )]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e77c87d-22de-4b6e-97c7-93101c133b72",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Statistics\n",
    "\n",
    "- No of xarray datasets\n",
    "\n",
    "    The number of xarray datasets per phase 1 simulation is equivalent to the sum of the entries in the intake catalogue for the specific simulation.\n",
    "    \n",
    "- No of variables \n",
    "\n",
    "    The number off variables per phase 1 simulation is computed by summing up `len(ds.data_vars)` for each xarray dataset `ds` of a simulation. That means, \"*variables*\" are a combination of *aggregation* and variable name, similar to the definition of a *CMOR variable*. 2m Temperature can be accounted multiple times if it is written for multiple datasets i.e. multiple aggregations.\n",
    "    \n",
    "- Size in memory [TB]\n",
    "\n",
    "    The size in memory per phase 1 simulation is computed by summing up `ds.size` for each xarray dataset `ds` of a simulation. This does not reflect the actual volume on disk because the datasets can be stored in a compressed form.\n",
    "    \n",
    "- Start simulation year\n",
    "\n",
    "    The start simulation year is the first year of the *example* dataset\n",
    "    \n",
    "- End simulation year\n",
    "\n",
    "    The start simulation year is the end year of the *example* dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4887fbb8-56f3-4752-a930-40699187ea46",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import intake\n",
    "eerie_cat=intake.open_catalog(\n",
    "    \"https://raw.githubusercontent.com/eerie-project/intake_catalogues/main/dkrz/disk/model-output/main.yaml\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "215240d1-ca94-4ee3-9e72-b4ff84867b1e",
   "metadata": {
    "jupyter": {
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "from copy import deepcopy\n",
    "def find_data_sources(catalog,name=None):\n",
    "    newname='.'.join(\n",
    "        [ a \n",
    "         for a in [name, catalog.name]\n",
    "         if a\n",
    "        ]\n",
    "    )\n",
    "    data_sources = []\n",
    "\n",
    "    for key, entry in catalog.items():\n",
    "        if isinstance(entry, intake.catalog.Catalog):\n",
    "            if newname == \"main\":\n",
    "                newname = None\n",
    "            # If the entry is a subcatalog, recursively search it\n",
    "            data_sources.extend(find_data_sources(entry, newname))\n",
    "        elif isinstance(entry, intake.source.base.DataSource):\n",
    "            if newname:\n",
    "                data_sources.append(newname+\".\"+key)\n",
    "            else:\n",
    "                data_sources.append(key)\n",
    "\n",
    "    return data_sources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e446b1ab-d7b3-47d4-98bd-20ec08a11eac",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true,
     "source_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ifs-fesom2-sr\n",
      "icon-esm-er\n",
      "ifs-amip\n",
      "ifs-nemo\n",
      "hadgem3-gc5-n640-orca12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siconc' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siu' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siv' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'mlotst10' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'sosabs' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'toscon' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'zos' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/indexing.py:1440: PerformanceWarning: Slicing is producing a large chunk. To accept the large\n",
      "chunk and silence this warning, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):\n",
      "    ...     array[indexer]\n",
      "\n",
      "To avoid creating the large chunks, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):\n",
      "    ...     array[indexer]\n",
      "  return self.array[key]\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/indexing.py:1440: PerformanceWarning: Slicing is producing a large chunk. To accept the large\n",
      "chunk and silence this warning, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):\n",
      "    ...     array[indexer]\n",
      "\n",
      "To avoid creating the large chunks, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):\n",
      "    ...     array[indexer]\n",
      "  return self.array[key]\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/indexing.py:1440: PerformanceWarning: Slicing is producing a large chunk. To accept the large\n",
      "chunk and silence this warning, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):\n",
      "    ...     array[indexer]\n",
      "\n",
      "To avoid creating the large chunks, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):\n",
      "    ...     array[indexer]\n",
      "  return self.array[key]\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/core/indexing.py:1440: PerformanceWarning: Slicing is producing a large chunk. To accept the large\n",
      "chunk and silence this warning, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': False}):\n",
      "    ...     array[indexer]\n",
      "\n",
      "To avoid creating the large chunks, set the option\n",
      "    >>> with dask.config.set(**{'array.slicing.split_large_chunks': True}):\n",
      "    ...     array[indexer]\n",
      "  return self.array[key]\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'mlotst10' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'sosabs' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'toscon' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'zos' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "hadgem3-gc5-n216-orca025\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siconc' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'sithick' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siu' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'siv' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'mlotst10' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'sosabs' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'toscon' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n",
      "/sw/spack-levante/mambaforge-23.1.0-1-Linux-x86_64-3boc6i/lib/python3.10/site-packages/xarray/conventions.py:427: SerializationWarning: variable 'zos' has multiple fill values {1e+20, 1e+20}, decoding all values to NaN.\n",
      "  new_vars[k] = decode_cf_variable(\n"
     ]
    }
   ],
   "source": [
    "from copy import deepcopy as copy\n",
    "dflist=[]\n",
    "for source_id,experiments in phase1_simulations.items():\n",
    "    print(source_id)\n",
    "    for idx,experiment in enumerate(experiments):\n",
    "        sdict=dict(source=source_id)\n",
    "        datasets={}        \n",
    "        cat_source=eerie_cat[source_id]\n",
    "        exp_id=experiment[\"experiment\"]        \n",
    "        sdict[\"experiment\"]=exp_id\n",
    "        version=experiment.get(\"version\",None)\n",
    "        sdict[\"version\"]=\"latest\"\n",
    "        cat_sourceexp=cat_source[exp_id]\n",
    "        if version:\n",
    "            sdict[\"version\"]=version[-1]\n",
    "            for vid in version:\n",
    "                cat_sourceexp=cat_sourceexp[vid]\n",
    "                for ds in find_data_sources(cat_sourceexp):\n",
    "                    datasets['.'.join([\n",
    "                        source_id,\n",
    "                        exp_id,\n",
    "                        ds\n",
    "                    ])]=cat_sourceexp['.'.join(ds.split('.')[1:])].to_dask()\n",
    "        else:\n",
    "            for ds in find_data_sources(cat_sourceexp):\n",
    "                datasets['.'.join([\n",
    "                    source_id,\n",
    "                    ds\n",
    "                ])]=cat_sourceexp['.'.join(ds.split('.')[1:])].to_dask()        \n",
    "        phase1_simulations[source_id][idx][\"datasets\"]=copy(datasets)\n",
    "        #\n",
    "        #assume datasets is the latest version\n",
    "        #\n",
    "        no_of_variables=0\n",
    "        size=0\n",
    "        sdict[\"No of xarray datasets\"]=len(datasets)\n",
    "        exds=None\n",
    "        for name,ds in datasets.items():\n",
    "            if experiment[\"example\"] in name:\n",
    "                exds=ds\n",
    "            if version:\n",
    "                if sdict[\"version\"] in name:\n",
    "                    size+=ds.nbytes\n",
    "                    no_of_variables+=len(ds.data_vars)\n",
    "            else:\n",
    "                size+=ds.nbytes\n",
    "                no_of_variables+=len(ds.data_vars)\n",
    "        sdict[\"No of variables\"]=no_of_variables\n",
    "        sdict[\"Size in memory [TB]\"]=size/1024**4\n",
    "        \n",
    "        years=exds[\"time\"].groupby(\"time.year\").groups        \n",
    "        sdict[\"Start simulation year\"]=list(years.keys())[0]\n",
    "        sdict[\"End simulation year\"]=list(years.keys())[-1]\n",
    "\n",
    "        dflist.append(copy(sdict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "87b9dc50-8e3b-4629-822d-bf26acdbd804",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "sourcedf=pd.DataFrame(dflist)#.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c708579a-4e83-469e-8920-972b0c4cc2c6",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>source</th>\n",
       "      <th>experiment</th>\n",
       "      <th>version</th>\n",
       "      <th>No of xarray datasets</th>\n",
       "      <th>No of variables</th>\n",
       "      <th>Size in memory [TB]</th>\n",
       "      <th>Start simulation year</th>\n",
       "      <th>End simulation year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ifs-fesom2-sr</td>\n",
       "      <td>eerie-spinup-1950</td>\n",
       "      <td>v20240304</td>\n",
       "      <td>12</td>\n",
       "      <td>86</td>\n",
       "      <td>31.303580</td>\n",
       "      <td>1950</td>\n",
       "      <td>1980</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>icon-esm-er</td>\n",
       "      <td>eerie-spinup-1950</td>\n",
       "      <td>v20240618</td>\n",
       "      <td>27</td>\n",
       "      <td>334</td>\n",
       "      <td>846.091654</td>\n",
       "      <td>1950</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>hadgem3-gc5-n640-orca12</td>\n",
       "      <td>eerie-picontrol</td>\n",
       "      <td>latest</td>\n",
       "      <td>9</td>\n",
       "      <td>117</td>\n",
       "      <td>3.413770</td>\n",
       "      <td>1851</td>\n",
       "      <td>1900</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>hadgem3-gc5-n216-orca025</td>\n",
       "      <td>eerie-picontrol</td>\n",
       "      <td>latest</td>\n",
       "      <td>9</td>\n",
       "      <td>115</td>\n",
       "      <td>1.671030</td>\n",
       "      <td>1851</td>\n",
       "      <td>1980</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     source         experiment    version   \n",
       "0             ifs-fesom2-sr  eerie-spinup-1950  v20240304  \\\n",
       "1               icon-esm-er  eerie-spinup-1950  v20240618   \n",
       "2   hadgem3-gc5-n640-orca12    eerie-picontrol     latest   \n",
       "3  hadgem3-gc5-n216-orca025    eerie-picontrol     latest   \n",
       "\n",
       "   No of xarray datasets  No of variables  Size in memory [TB]   \n",
       "0                     12               86            31.303580  \\\n",
       "1                     27              334           846.091654   \n",
       "2                      9              117             3.413770   \n",
       "3                      9              115             1.671030   \n",
       "\n",
       "   Start simulation year  End simulation year  \n",
       "0                   1950                 1980  \n",
       "1                   1950                 2000  \n",
       "2                   1851                 1900  \n",
       "3                   1851                 1980  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sourcedf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e19622f-7bac-41b2-bcf1-44c73a70ff56",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "0 Python 3 (based on the module python3/unstable",
   "language": "python",
   "name": "python3_unstable"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}