diff --git a/notebooks/use-intake-esgf-with-rooki.ipynb b/notebooks/use-intake-esgf-with-rooki.ipynb new file mode 100644 index 0000000..3f508c2 --- /dev/null +++ b/notebooks/use-intake-esgf-with-rooki.ipynb @@ -0,0 +1,2088 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "5191269c-944c-4516-9f51-6cdfc704852a", + "metadata": {}, + "source": [ + "\"Intake" + ] + }, + { + "cell_type": "markdown", + "id": "fa96801d-4d1a-4264-94f9-9bf12a77421a", + "metadata": {}, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "id": "545103ee-abac-4da8-af3b-c8877a3d2d6c", + "metadata": {}, + "source": [ + "# Using intake-esgf with rooki\n", + "Here we dig into using intake-esgf to search for data, then rooki to do server-side computing!" + ] + }, + { + "cell_type": "markdown", + "id": "7f90a23b-fd23-4339-924f-d665f1d36472", + "metadata": {}, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "id": "d57fa890-a874-4282-bdf2-dfb905678467", + "metadata": {}, + "source": [ + "## Overview\n", + "If you have an introductory paragraph, lead with it here! Keep it short and tied to your material, then be sure to continue into the required list of topics below,\n", + "\n", + "1. Search and find data using intake-esgf, returning the dataset ids\n", + "1. Feed the dataset ids to rooki to subset and average the data remotely\n", + "1. Visualize the results on the end-user side" + ] + }, + { + "cell_type": "markdown", + "id": "0b633dab-4c9d-482a-b148-8f9b09102e78", + "metadata": {}, + "source": [ + "## Prerequisites\n", + "\n", + "| Concepts | Importance | Notes |\n", + "| --- | --- | --- |\n", + "| [Intro to Intake-ESGF](intro-search) | Necessary | How to configure a search and use output |\n", + "| [Intro to Rooki](rooki) | Helpful | How to initialize and run rooki |\n", + "| [Intro to hvPlot](https://hvplot.holoviz.org/user_guide/Geographic_Data.html) | Necessary | How to plot interactive visualizations |\n", + "\n", + "- **Time to learn**: 30 minutes" + ] + }, + { + "cell_type": "markdown", + "id": "64a5b6f9-eeee-4726-abdc-686e96dfc3cf", + "metadata": {}, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "id": "f7571ba4-43cf-4f61-b74a-a51cdee373aa", + "metadata": {}, + "source": [ + "## Imports" + ] + }, + { + "cell_type": "code", + "execution_count": 89, + "id": "c12d0875-9794-4101-b74f-346148cb36c0", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "application/javascript": [ + "(function(root) {\n", + " function now() {\n", + " return new Date();\n", + " }\n", + "\n", + " var force = true;\n", + " var py_version = '3.4.0'.replace('rc', '-rc.').replace('.dev', '-dev.');\n", + " var reloading = false;\n", + " var Bokeh = root.Bokeh;\n", + "\n", + " if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n", + " root._bokeh_timeout = Date.now() + 5000;\n", + " root._bokeh_failed_load = false;\n", + " }\n", + "\n", + " function run_callbacks() {\n", + " try {\n", + " root._bokeh_onload_callbacks.forEach(function(callback) {\n", + " if (callback != null)\n", + " callback();\n", + " });\n", + " } finally {\n", + " delete root._bokeh_onload_callbacks;\n", + " }\n", + " console.debug(\"Bokeh: all callbacks have finished\");\n", + " }\n", + "\n", + " function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n", + " if (css_urls == null) css_urls = [];\n", + " if (js_urls == null) js_urls = [];\n", + " if (js_modules == null) js_modules = [];\n", + " if (js_exports == null) js_exports = {};\n", + "\n", + " root._bokeh_onload_callbacks.push(callback);\n", + "\n", + " if (root._bokeh_is_loading > 0) {\n", + " console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n", + " return null;\n", + " }\n", + " if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n", + " run_callbacks();\n", + " return null;\n", + " }\n", + " if (!reloading) {\n", + " console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n", + " }\n", + "\n", + " function on_load() {\n", + " root._bokeh_is_loading--;\n", + " if (root._bokeh_is_loading === 0) {\n", + " console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n", + " run_callbacks()\n", + " }\n", + " }\n", + " window._bokeh_on_load = on_load\n", + "\n", + " function on_error() {\n", + " console.error(\"failed to load \" + url);\n", + " }\n", + "\n", + " var skip = [];\n", + " if (window.requirejs) {\n", + " window.requirejs.config({'packages': {}, 'paths': {}, 'shim': {}});\n", + " root._bokeh_is_loading = css_urls.length + 0;\n", + " } else {\n", + " root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n", + " }\n", + "\n", + " var existing_stylesheets = []\n", + " var links = document.getElementsByTagName('link')\n", + " for (var i = 0; i < links.length; i++) {\n", + " var link = links[i]\n", + " if (link.href != null) {\n", + "\texisting_stylesheets.push(link.href)\n", + " }\n", + " }\n", + " for (var i = 0; i < css_urls.length; i++) {\n", + " var url = css_urls[i];\n", + " if (existing_stylesheets.indexOf(url) !== -1) {\n", + "\ton_load()\n", + "\tcontinue;\n", + " }\n", + " const element = document.createElement(\"link\");\n", + " element.onload = on_load;\n", + " element.onerror = on_error;\n", + " element.rel = \"stylesheet\";\n", + " element.type = \"text/css\";\n", + " element.href = url;\n", + " console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n", + " document.body.appendChild(element);\n", + " } var existing_scripts = []\n", + " var scripts = document.getElementsByTagName('script')\n", + " for (var i = 0; i < scripts.length; i++) {\n", + " var script = scripts[i]\n", + " if (script.src != null) {\n", + "\texisting_scripts.push(script.src)\n", + " }\n", + " }\n", + " for (var i = 0; i < js_urls.length; i++) {\n", + " var url = js_urls[i];\n", + " if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n", + "\tif (!window.requirejs) {\n", + "\t on_load();\n", + "\t}\n", + "\tcontinue;\n", + " }\n", + " var element = document.createElement('script');\n", + " element.onload = on_load;\n", + " element.onerror = on_error;\n", + " element.async = false;\n", + " element.src = url;\n", + " console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n", + " document.head.appendChild(element);\n", + " }\n", + " for (var i = 0; i < js_modules.length; i++) {\n", + " var url = js_modules[i];\n", + " if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n", + "\tif (!window.requirejs) {\n", + "\t on_load();\n", + "\t}\n", + "\tcontinue;\n", + " }\n", + " var element = document.createElement('script');\n", + " element.onload = on_load;\n", + " element.onerror = on_error;\n", + " element.async = false;\n", + " element.src = url;\n", + " element.type = \"module\";\n", + " console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n", + " document.head.appendChild(element);\n", + " }\n", + " for (const name in js_exports) {\n", + " var url = js_exports[name];\n", + " if (skip.indexOf(url) >= 0 || root[name] != null) {\n", + "\tif (!window.requirejs) {\n", + "\t on_load();\n", + "\t}\n", + "\tcontinue;\n", + " }\n", + " var element = document.createElement('script');\n", + " element.onerror = on_error;\n", + " element.async = false;\n", + " element.type = \"module\";\n", + " console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n", + " element.textContent = `\n", + " import ${name} from \"${url}\"\n", + " window.${name} = ${name}\n", + " window._bokeh_on_load()\n", + " `\n", + " document.head.appendChild(element);\n", + " }\n", + " if (!js_urls.length && !js_modules.length) {\n", + " on_load()\n", + " }\n", + " };\n", + "\n", + " function inject_raw_css(css) {\n", + " const element = document.createElement(\"style\");\n", + " element.appendChild(document.createTextNode(css));\n", + " document.body.appendChild(element);\n", + " }\n", + "\n", + " var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.4.0.min.js\", \"https://cdn.holoviz.org/panel/1.4.1/dist/panel.min.js\", \"https://cdn.jsdelivr.net/npm/@holoviz/geoviews@1.12.0/dist/geoviews.min.js\"];\n", + " var js_modules = [];\n", + " var js_exports = {};\n", + " var css_urls = [];\n", + " var inline_js = [ function(Bokeh) {\n", + " Bokeh.set_log_level(\"info\");\n", + " },\n", + "function(Bokeh) {} // ensure no trailing comma for IE\n", + " ];\n", + "\n", + " function run_inline_js() {\n", + " if ((root.Bokeh !== undefined) || (force === true)) {\n", + " for (var i = 0; i < inline_js.length; i++) {\n", + "\ttry {\n", + " inline_js[i].call(root, root.Bokeh);\n", + "\t} catch(e) {\n", + "\t if (!reloading) {\n", + "\t throw e;\n", + "\t }\n", + "\t}\n", + " }\n", + " // Cache old bokeh versions\n", + " if (Bokeh != undefined && !reloading) {\n", + "\tvar NewBokeh = root.Bokeh;\n", + "\tif (Bokeh.versions === undefined) {\n", + "\t Bokeh.versions = new Map();\n", + "\t}\n", + "\tif (NewBokeh.version !== Bokeh.version) {\n", + "\t Bokeh.versions.set(NewBokeh.version, NewBokeh)\n", + "\t}\n", + "\troot.Bokeh = Bokeh;\n", + " }} else if (Date.now() < root._bokeh_timeout) {\n", + " setTimeout(run_inline_js, 100);\n", + " } else if (!root._bokeh_failed_load) {\n", + " console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n", + " root._bokeh_failed_load = true;\n", + " }\n", + " root._bokeh_is_initializing = false\n", + " }\n", + "\n", + " function load_or_wait() {\n", + " // Implement a backoff loop that tries to ensure we do not load multiple\n", + " // versions of Bokeh and its dependencies at the same time.\n", + " // In recent versions we use the root._bokeh_is_initializing flag\n", + " // to determine whether there is an ongoing attempt to initialize\n", + " // bokeh, however for backward compatibility we also try to ensure\n", + " // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n", + " // before older versions are fully initialized.\n", + " if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n", + " root._bokeh_is_initializing = false;\n", + " root._bokeh_onload_callbacks = undefined;\n", + " console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n", + " load_or_wait();\n", + " } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n", + " setTimeout(load_or_wait, 100);\n", + " } else {\n", + " root._bokeh_is_initializing = true\n", + " root._bokeh_onload_callbacks = []\n", + " var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n", + " if (!reloading && !bokeh_loaded) {\n", + "\troot.Bokeh = undefined;\n", + " }\n", + " load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n", + "\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n", + "\trun_inline_js();\n", + " });\n", + " }\n", + " }\n", + " // Give older versions of the autoload script a head-start to ensure\n", + " // they initialize before we start loading newer version.\n", + " setTimeout(load_or_wait, 100)\n", + "}(window));" + ], + "application/vnd.holoviews_load.v0+json": "(function(root) {\n function now() {\n return new Date();\n }\n\n var force = true;\n var py_version = '3.4.0'.replace('rc', '-rc.').replace('.dev', '-dev.');\n var reloading = false;\n var Bokeh = root.Bokeh;\n\n if (typeof (root._bokeh_timeout) === \"undefined\" || force) {\n root._bokeh_timeout = Date.now() + 5000;\n root._bokeh_failed_load = false;\n }\n\n function run_callbacks() {\n try {\n root._bokeh_onload_callbacks.forEach(function(callback) {\n if (callback != null)\n callback();\n });\n } finally {\n delete root._bokeh_onload_callbacks;\n }\n console.debug(\"Bokeh: all callbacks have finished\");\n }\n\n function load_libs(css_urls, js_urls, js_modules, js_exports, callback) {\n if (css_urls == null) css_urls = [];\n if (js_urls == null) js_urls = [];\n if (js_modules == null) js_modules = [];\n if (js_exports == null) js_exports = {};\n\n root._bokeh_onload_callbacks.push(callback);\n\n if (root._bokeh_is_loading > 0) {\n console.debug(\"Bokeh: BokehJS is being loaded, scheduling callback at\", now());\n return null;\n }\n if (js_urls.length === 0 && js_modules.length === 0 && Object.keys(js_exports).length === 0) {\n run_callbacks();\n return null;\n }\n if (!reloading) {\n console.debug(\"Bokeh: BokehJS not loaded, scheduling load and callback at\", now());\n }\n\n function on_load() {\n root._bokeh_is_loading--;\n if (root._bokeh_is_loading === 0) {\n console.debug(\"Bokeh: all BokehJS libraries/stylesheets loaded\");\n run_callbacks()\n }\n }\n window._bokeh_on_load = on_load\n\n function on_error() {\n console.error(\"failed to load \" + url);\n }\n\n var skip = [];\n if (window.requirejs) {\n window.requirejs.config({'packages': {}, 'paths': {}, 'shim': {}});\n root._bokeh_is_loading = css_urls.length + 0;\n } else {\n root._bokeh_is_loading = css_urls.length + js_urls.length + js_modules.length + Object.keys(js_exports).length;\n }\n\n var existing_stylesheets = []\n var links = document.getElementsByTagName('link')\n for (var i = 0; i < links.length; i++) {\n var link = links[i]\n if (link.href != null) {\n\texisting_stylesheets.push(link.href)\n }\n }\n for (var i = 0; i < css_urls.length; i++) {\n var url = css_urls[i];\n if (existing_stylesheets.indexOf(url) !== -1) {\n\ton_load()\n\tcontinue;\n }\n const element = document.createElement(\"link\");\n element.onload = on_load;\n element.onerror = on_error;\n element.rel = \"stylesheet\";\n element.type = \"text/css\";\n element.href = url;\n console.debug(\"Bokeh: injecting link tag for BokehJS stylesheet: \", url);\n document.body.appendChild(element);\n } var existing_scripts = []\n var scripts = document.getElementsByTagName('script')\n for (var i = 0; i < scripts.length; i++) {\n var script = scripts[i]\n if (script.src != null) {\n\texisting_scripts.push(script.src)\n }\n }\n for (var i = 0; i < js_urls.length; i++) {\n var url = js_urls[i];\n if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t on_load();\n\t}\n\tcontinue;\n }\n var element = document.createElement('script');\n element.onload = on_load;\n element.onerror = on_error;\n element.async = false;\n element.src = url;\n console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n document.head.appendChild(element);\n }\n for (var i = 0; i < js_modules.length; i++) {\n var url = js_modules[i];\n if (skip.indexOf(url) !== -1 || existing_scripts.indexOf(url) !== -1) {\n\tif (!window.requirejs) {\n\t on_load();\n\t}\n\tcontinue;\n }\n var element = document.createElement('script');\n element.onload = on_load;\n element.onerror = on_error;\n element.async = false;\n element.src = url;\n element.type = \"module\";\n console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n document.head.appendChild(element);\n }\n for (const name in js_exports) {\n var url = js_exports[name];\n if (skip.indexOf(url) >= 0 || root[name] != null) {\n\tif (!window.requirejs) {\n\t on_load();\n\t}\n\tcontinue;\n }\n var element = document.createElement('script');\n element.onerror = on_error;\n element.async = false;\n element.type = \"module\";\n console.debug(\"Bokeh: injecting script tag for BokehJS library: \", url);\n element.textContent = `\n import ${name} from \"${url}\"\n window.${name} = ${name}\n window._bokeh_on_load()\n `\n document.head.appendChild(element);\n }\n if (!js_urls.length && !js_modules.length) {\n on_load()\n }\n };\n\n function inject_raw_css(css) {\n const element = document.createElement(\"style\");\n element.appendChild(document.createTextNode(css));\n document.body.appendChild(element);\n }\n\n var js_urls = [\"https://cdn.bokeh.org/bokeh/release/bokeh-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-gl-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-widgets-3.4.0.min.js\", \"https://cdn.bokeh.org/bokeh/release/bokeh-tables-3.4.0.min.js\", \"https://cdn.holoviz.org/panel/1.4.1/dist/panel.min.js\", \"https://cdn.jsdelivr.net/npm/@holoviz/geoviews@1.12.0/dist/geoviews.min.js\"];\n var js_modules = [];\n var js_exports = {};\n var css_urls = [];\n var inline_js = [ function(Bokeh) {\n Bokeh.set_log_level(\"info\");\n },\nfunction(Bokeh) {} // ensure no trailing comma for IE\n ];\n\n function run_inline_js() {\n if ((root.Bokeh !== undefined) || (force === true)) {\n for (var i = 0; i < inline_js.length; i++) {\n\ttry {\n inline_js[i].call(root, root.Bokeh);\n\t} catch(e) {\n\t if (!reloading) {\n\t throw e;\n\t }\n\t}\n }\n // Cache old bokeh versions\n if (Bokeh != undefined && !reloading) {\n\tvar NewBokeh = root.Bokeh;\n\tif (Bokeh.versions === undefined) {\n\t Bokeh.versions = new Map();\n\t}\n\tif (NewBokeh.version !== Bokeh.version) {\n\t Bokeh.versions.set(NewBokeh.version, NewBokeh)\n\t}\n\troot.Bokeh = Bokeh;\n }} else if (Date.now() < root._bokeh_timeout) {\n setTimeout(run_inline_js, 100);\n } else if (!root._bokeh_failed_load) {\n console.log(\"Bokeh: BokehJS failed to load within specified timeout.\");\n root._bokeh_failed_load = true;\n }\n root._bokeh_is_initializing = false\n }\n\n function load_or_wait() {\n // Implement a backoff loop that tries to ensure we do not load multiple\n // versions of Bokeh and its dependencies at the same time.\n // In recent versions we use the root._bokeh_is_initializing flag\n // to determine whether there is an ongoing attempt to initialize\n // bokeh, however for backward compatibility we also try to ensure\n // that we do not start loading a newer (Panel>=1.0 and Bokeh>3) version\n // before older versions are fully initialized.\n if (root._bokeh_is_initializing && Date.now() > root._bokeh_timeout) {\n root._bokeh_is_initializing = false;\n root._bokeh_onload_callbacks = undefined;\n console.log(\"Bokeh: BokehJS was loaded multiple times but one version failed to initialize.\");\n load_or_wait();\n } else if (root._bokeh_is_initializing || (typeof root._bokeh_is_initializing === \"undefined\" && root._bokeh_onload_callbacks !== undefined)) {\n setTimeout(load_or_wait, 100);\n } else {\n root._bokeh_is_initializing = true\n root._bokeh_onload_callbacks = []\n var bokeh_loaded = Bokeh != null && (Bokeh.version === py_version || (Bokeh.versions !== undefined && Bokeh.versions.has(py_version)));\n if (!reloading && !bokeh_loaded) {\n\troot.Bokeh = undefined;\n }\n load_libs(css_urls, js_urls, js_modules, js_exports, function() {\n\tconsole.debug(\"Bokeh: BokehJS plotting callback run at\", now());\n\trun_inline_js();\n });\n }\n }\n // Give older versions of the autoload script a head-start to ensure\n // they initialize before we start loading newer version.\n setTimeout(load_or_wait, 100)\n}(window));" + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/javascript": [ + "\n", + "if ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n", + " window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n", + "}\n", + "\n", + "\n", + " function JupyterCommManager() {\n", + " }\n", + "\n", + " JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n", + " if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n", + " var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n", + " comm_manager.register_target(comm_id, function(comm) {\n", + " comm.on_msg(msg_handler);\n", + " });\n", + " } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n", + " window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n", + " comm.onMsg = msg_handler;\n", + " });\n", + " } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n", + " google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n", + " var messages = comm.messages[Symbol.asyncIterator]();\n", + " function processIteratorResult(result) {\n", + " var message = result.value;\n", + " console.log(message)\n", + " var content = {data: message.data, comm_id};\n", + " var buffers = []\n", + " for (var buffer of message.buffers || []) {\n", + " buffers.push(new DataView(buffer))\n", + " }\n", + " var metadata = message.metadata || {};\n", + " var msg = {content, buffers, metadata}\n", + " msg_handler(msg);\n", + " return messages.next().then(processIteratorResult);\n", + " }\n", + " return messages.next().then(processIteratorResult);\n", + " })\n", + " }\n", + " }\n", + "\n", + " JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n", + " if (comm_id in window.PyViz.comms) {\n", + " return window.PyViz.comms[comm_id];\n", + " } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n", + " var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n", + " var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n", + " if (msg_handler) {\n", + " comm.on_msg(msg_handler);\n", + " }\n", + " } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n", + " var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n", + " comm.open();\n", + " if (msg_handler) {\n", + " comm.onMsg = msg_handler;\n", + " }\n", + " } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n", + " var comm_promise = google.colab.kernel.comms.open(comm_id)\n", + " comm_promise.then((comm) => {\n", + " window.PyViz.comms[comm_id] = comm;\n", + " if (msg_handler) {\n", + " var messages = comm.messages[Symbol.asyncIterator]();\n", + " function processIteratorResult(result) {\n", + " var message = result.value;\n", + " var content = {data: message.data};\n", + " var metadata = message.metadata || {comm_id};\n", + " var msg = {content, metadata}\n", + " msg_handler(msg);\n", + " return messages.next().then(processIteratorResult);\n", + " }\n", + " return messages.next().then(processIteratorResult);\n", + " }\n", + " }) \n", + " var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n", + " return comm_promise.then((comm) => {\n", + " comm.send(data, metadata, buffers, disposeOnDone);\n", + " });\n", + " };\n", + " var comm = {\n", + " send: sendClosure\n", + " };\n", + " }\n", + " window.PyViz.comms[comm_id] = comm;\n", + " return comm;\n", + " }\n", + " window.PyViz.comm_manager = new JupyterCommManager();\n", + " \n", + "\n", + "\n", + "var JS_MIME_TYPE = 'application/javascript';\n", + "var HTML_MIME_TYPE = 'text/html';\n", + "var EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\n", + "var CLASS_NAME = 'output';\n", + "\n", + "/**\n", + " * Render data to the DOM node\n", + " */\n", + "function render(props, node) {\n", + " var div = document.createElement(\"div\");\n", + " var script = document.createElement(\"script\");\n", + " node.appendChild(div);\n", + " node.appendChild(script);\n", + "}\n", + "\n", + "/**\n", + " * Handle when a new output is added\n", + " */\n", + "function handle_add_output(event, handle) {\n", + " var output_area = handle.output_area;\n", + " var output = handle.output;\n", + " if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n", + " return\n", + " }\n", + " var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n", + " var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n", + " if (id !== undefined) {\n", + " var nchildren = toinsert.length;\n", + " var html_node = toinsert[nchildren-1].children[0];\n", + " html_node.innerHTML = output.data[HTML_MIME_TYPE];\n", + " var scripts = [];\n", + " var nodelist = html_node.querySelectorAll(\"script\");\n", + " for (var i in nodelist) {\n", + " if (nodelist.hasOwnProperty(i)) {\n", + " scripts.push(nodelist[i])\n", + " }\n", + " }\n", + "\n", + " scripts.forEach( function (oldScript) {\n", + " var newScript = document.createElement(\"script\");\n", + " var attrs = [];\n", + " var nodemap = oldScript.attributes;\n", + " for (var j in nodemap) {\n", + " if (nodemap.hasOwnProperty(j)) {\n", + " attrs.push(nodemap[j])\n", + " }\n", + " }\n", + " attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n", + " newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n", + " oldScript.parentNode.replaceChild(newScript, oldScript);\n", + " });\n", + " if (JS_MIME_TYPE in output.data) {\n", + " toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n", + " }\n", + " output_area._hv_plot_id = id;\n", + " if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n", + " window.PyViz.plot_index[id] = Bokeh.index[id];\n", + " } else {\n", + " window.PyViz.plot_index[id] = null;\n", + " }\n", + " } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n", + " var bk_div = document.createElement(\"div\");\n", + " bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n", + " var script_attrs = bk_div.children[0].attributes;\n", + " for (var i = 0; i < script_attrs.length; i++) {\n", + " toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n", + " }\n", + " // store reference to server id on output_area\n", + " output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n", + " }\n", + "}\n", + "\n", + "/**\n", + " * Handle when an output is cleared or removed\n", + " */\n", + "function handle_clear_output(event, handle) {\n", + " var id = handle.cell.output_area._hv_plot_id;\n", + " var server_id = handle.cell.output_area._bokeh_server_id;\n", + " if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n", + " var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n", + " if (server_id !== null) {\n", + " comm.send({event_type: 'server_delete', 'id': server_id});\n", + " return;\n", + " } else if (comm !== null) {\n", + " comm.send({event_type: 'delete', 'id': id});\n", + " }\n", + " delete PyViz.plot_index[id];\n", + " if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n", + " var doc = window.Bokeh.index[id].model.document\n", + " doc.clear();\n", + " const i = window.Bokeh.documents.indexOf(doc);\n", + " if (i > -1) {\n", + " window.Bokeh.documents.splice(i, 1);\n", + " }\n", + " }\n", + "}\n", + "\n", + "/**\n", + " * Handle kernel restart event\n", + " */\n", + "function handle_kernel_cleanup(event, handle) {\n", + " delete PyViz.comms[\"hv-extension-comm\"];\n", + " window.PyViz.plot_index = {}\n", + "}\n", + "\n", + "/**\n", + " * Handle update_display_data messages\n", + " */\n", + "function handle_update_output(event, handle) {\n", + " handle_clear_output(event, {cell: {output_area: handle.output_area}})\n", + " handle_add_output(event, handle)\n", + "}\n", + "\n", + "function register_renderer(events, OutputArea) {\n", + " function append_mime(data, metadata, element) {\n", + " // create a DOM node to render to\n", + " var toinsert = this.create_output_subarea(\n", + " metadata,\n", + " CLASS_NAME,\n", + " EXEC_MIME_TYPE\n", + " );\n", + " this.keyboard_manager.register_events(toinsert);\n", + " // Render to node\n", + " var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n", + " render(props, toinsert[0]);\n", + " element.append(toinsert);\n", + " return toinsert\n", + " }\n", + "\n", + " events.on('output_added.OutputArea', handle_add_output);\n", + " events.on('output_updated.OutputArea', handle_update_output);\n", + " events.on('clear_output.CodeCell', handle_clear_output);\n", + " events.on('delete.Cell', handle_clear_output);\n", + " events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n", + "\n", + " OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n", + " safe: true,\n", + " index: 0\n", + " });\n", + "}\n", + "\n", + "if (window.Jupyter !== undefined) {\n", + " try {\n", + " var events = require('base/js/events');\n", + " var OutputArea = require('notebook/js/outputarea').OutputArea;\n", + " if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n", + " register_renderer(events, OutputArea);\n", + " }\n", + " } catch(err) {\n", + " }\n", + "}\n" + ], + "application/vnd.holoviews_load.v0+json": "\nif ((window.PyViz === undefined) || (window.PyViz instanceof HTMLElement)) {\n window.PyViz = {comms: {}, comm_status:{}, kernels:{}, receivers: {}, plot_index: []}\n}\n\n\n function JupyterCommManager() {\n }\n\n JupyterCommManager.prototype.register_target = function(plot_id, comm_id, msg_handler) {\n if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n comm_manager.register_target(comm_id, function(comm) {\n comm.on_msg(msg_handler);\n });\n } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n window.PyViz.kernels[plot_id].registerCommTarget(comm_id, function(comm) {\n comm.onMsg = msg_handler;\n });\n } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n google.colab.kernel.comms.registerTarget(comm_id, (comm) => {\n var messages = comm.messages[Symbol.asyncIterator]();\n function processIteratorResult(result) {\n var message = result.value;\n console.log(message)\n var content = {data: message.data, comm_id};\n var buffers = []\n for (var buffer of message.buffers || []) {\n buffers.push(new DataView(buffer))\n }\n var metadata = message.metadata || {};\n var msg = {content, buffers, metadata}\n msg_handler(msg);\n return messages.next().then(processIteratorResult);\n }\n return messages.next().then(processIteratorResult);\n })\n }\n }\n\n JupyterCommManager.prototype.get_client_comm = function(plot_id, comm_id, msg_handler) {\n if (comm_id in window.PyViz.comms) {\n return window.PyViz.comms[comm_id];\n } else if (window.comm_manager || ((window.Jupyter !== undefined) && (Jupyter.notebook.kernel != null))) {\n var comm_manager = window.comm_manager || Jupyter.notebook.kernel.comm_manager;\n var comm = comm_manager.new_comm(comm_id, {}, {}, {}, comm_id);\n if (msg_handler) {\n comm.on_msg(msg_handler);\n }\n } else if ((plot_id in window.PyViz.kernels) && (window.PyViz.kernels[plot_id])) {\n var comm = window.PyViz.kernels[plot_id].connectToComm(comm_id);\n comm.open();\n if (msg_handler) {\n comm.onMsg = msg_handler;\n }\n } else if (typeof google != 'undefined' && google.colab.kernel != null) {\n var comm_promise = google.colab.kernel.comms.open(comm_id)\n comm_promise.then((comm) => {\n window.PyViz.comms[comm_id] = comm;\n if (msg_handler) {\n var messages = comm.messages[Symbol.asyncIterator]();\n function processIteratorResult(result) {\n var message = result.value;\n var content = {data: message.data};\n var metadata = message.metadata || {comm_id};\n var msg = {content, metadata}\n msg_handler(msg);\n return messages.next().then(processIteratorResult);\n }\n return messages.next().then(processIteratorResult);\n }\n }) \n var sendClosure = (data, metadata, buffers, disposeOnDone) => {\n return comm_promise.then((comm) => {\n comm.send(data, metadata, buffers, disposeOnDone);\n });\n };\n var comm = {\n send: sendClosure\n };\n }\n window.PyViz.comms[comm_id] = comm;\n return comm;\n }\n window.PyViz.comm_manager = new JupyterCommManager();\n \n\n\nvar JS_MIME_TYPE = 'application/javascript';\nvar HTML_MIME_TYPE = 'text/html';\nvar EXEC_MIME_TYPE = 'application/vnd.holoviews_exec.v0+json';\nvar CLASS_NAME = 'output';\n\n/**\n * Render data to the DOM node\n */\nfunction render(props, node) {\n var div = document.createElement(\"div\");\n var script = document.createElement(\"script\");\n node.appendChild(div);\n node.appendChild(script);\n}\n\n/**\n * Handle when a new output is added\n */\nfunction handle_add_output(event, handle) {\n var output_area = handle.output_area;\n var output = handle.output;\n if ((output.data == undefined) || (!output.data.hasOwnProperty(EXEC_MIME_TYPE))) {\n return\n }\n var id = output.metadata[EXEC_MIME_TYPE][\"id\"];\n var toinsert = output_area.element.find(\".\" + CLASS_NAME.split(' ')[0]);\n if (id !== undefined) {\n var nchildren = toinsert.length;\n var html_node = toinsert[nchildren-1].children[0];\n html_node.innerHTML = output.data[HTML_MIME_TYPE];\n var scripts = [];\n var nodelist = html_node.querySelectorAll(\"script\");\n for (var i in nodelist) {\n if (nodelist.hasOwnProperty(i)) {\n scripts.push(nodelist[i])\n }\n }\n\n scripts.forEach( function (oldScript) {\n var newScript = document.createElement(\"script\");\n var attrs = [];\n var nodemap = oldScript.attributes;\n for (var j in nodemap) {\n if (nodemap.hasOwnProperty(j)) {\n attrs.push(nodemap[j])\n }\n }\n attrs.forEach(function(attr) { newScript.setAttribute(attr.name, attr.value) });\n newScript.appendChild(document.createTextNode(oldScript.innerHTML));\n oldScript.parentNode.replaceChild(newScript, oldScript);\n });\n if (JS_MIME_TYPE in output.data) {\n toinsert[nchildren-1].children[1].textContent = output.data[JS_MIME_TYPE];\n }\n output_area._hv_plot_id = id;\n if ((window.Bokeh !== undefined) && (id in Bokeh.index)) {\n window.PyViz.plot_index[id] = Bokeh.index[id];\n } else {\n window.PyViz.plot_index[id] = null;\n }\n } else if (output.metadata[EXEC_MIME_TYPE][\"server_id\"] !== undefined) {\n var bk_div = document.createElement(\"div\");\n bk_div.innerHTML = output.data[HTML_MIME_TYPE];\n var script_attrs = bk_div.children[0].attributes;\n for (var i = 0; i < script_attrs.length; i++) {\n toinsert[toinsert.length - 1].childNodes[1].setAttribute(script_attrs[i].name, script_attrs[i].value);\n }\n // store reference to server id on output_area\n output_area._bokeh_server_id = output.metadata[EXEC_MIME_TYPE][\"server_id\"];\n }\n}\n\n/**\n * Handle when an output is cleared or removed\n */\nfunction handle_clear_output(event, handle) {\n var id = handle.cell.output_area._hv_plot_id;\n var server_id = handle.cell.output_area._bokeh_server_id;\n if (((id === undefined) || !(id in PyViz.plot_index)) && (server_id !== undefined)) { return; }\n var comm = window.PyViz.comm_manager.get_client_comm(\"hv-extension-comm\", \"hv-extension-comm\", function () {});\n if (server_id !== null) {\n comm.send({event_type: 'server_delete', 'id': server_id});\n return;\n } else if (comm !== null) {\n comm.send({event_type: 'delete', 'id': id});\n }\n delete PyViz.plot_index[id];\n if ((window.Bokeh !== undefined) & (id in window.Bokeh.index)) {\n var doc = window.Bokeh.index[id].model.document\n doc.clear();\n const i = window.Bokeh.documents.indexOf(doc);\n if (i > -1) {\n window.Bokeh.documents.splice(i, 1);\n }\n }\n}\n\n/**\n * Handle kernel restart event\n */\nfunction handle_kernel_cleanup(event, handle) {\n delete PyViz.comms[\"hv-extension-comm\"];\n window.PyViz.plot_index = {}\n}\n\n/**\n * Handle update_display_data messages\n */\nfunction handle_update_output(event, handle) {\n handle_clear_output(event, {cell: {output_area: handle.output_area}})\n handle_add_output(event, handle)\n}\n\nfunction register_renderer(events, OutputArea) {\n function append_mime(data, metadata, element) {\n // create a DOM node to render to\n var toinsert = this.create_output_subarea(\n metadata,\n CLASS_NAME,\n EXEC_MIME_TYPE\n );\n this.keyboard_manager.register_events(toinsert);\n // Render to node\n var props = {data: data, metadata: metadata[EXEC_MIME_TYPE]};\n render(props, toinsert[0]);\n element.append(toinsert);\n return toinsert\n }\n\n events.on('output_added.OutputArea', handle_add_output);\n events.on('output_updated.OutputArea', handle_update_output);\n events.on('clear_output.CodeCell', handle_clear_output);\n events.on('delete.Cell', handle_clear_output);\n events.on('kernel_ready.Kernel', handle_kernel_cleanup);\n\n OutputArea.prototype.register_mime_type(EXEC_MIME_TYPE, append_mime, {\n safe: true,\n index: 0\n });\n}\n\nif (window.Jupyter !== undefined) {\n try {\n var events = require('base/js/events');\n var OutputArea = require('notebook/js/outputarea').OutputArea;\n if (OutputArea.prototype.mime_types().indexOf(EXEC_MIME_TYPE) == -1) {\n register_renderer(events, OutputArea);\n }\n } catch(err) {\n }\n}\n" + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.holoviews_exec.v0+json": "", + "text/html": [ + "
\n", + "
\n", + "
\n", + "" + ] + }, + "metadata": { + "application/vnd.holoviews_exec.v0+json": { + "id": "p13376" + } + }, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "\n", + "
\n", + "\n", + "\n", + "\n", + " \n", + " \n", + "\n", + "\n", + "\n", + "\n", + "
\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import os\n", + "\n", + "from rooki import rooki\n", + "from rooki import operators as ops\n", + "import intake_esgf\n", + "from intake_esgf import ESGFCatalog\n", + "import xarray as xr\n", + "import hvplot.xarray\n", + "import holoviews as hv\n", + "import panel as pn\n", + "hv.extension(\"bokeh\")" + ] + }, + { + "cell_type": "markdown", + "id": "7dd9b81a-9aeb-4769-8c67-4db88a089859", + "metadata": {}, + "source": [ + "## Search and Find Data for Surface Temperature on DKRZ Node\n", + "\n", + "Let's start with refining which index we would like to search from. For this analysis, we are remotely computing on the DKRZ node since this is where rooki is running. We know this from checking the `._url` method of rooki!" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "d3cdea76-9a74-4001-91b4-6c1b664835b2", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'http://rook.dkrz.de/wps'" + ] + }, + "execution_count": 18, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "rooki._url" + ] + }, + { + "cell_type": "markdown", + "id": "4a4442f9-6c41-4a56-ac66-4299d0c4ddda", + "metadata": {}, + "source": [ + "### Set the Index Node and Search\n", + "We need to turn off the new index nodes (Argonne National Lab (`anl-dev`) and Oak Ridge National Lab (`ornl-dev`)), and turn on the German Climate Computation Center (`esgf-data.dkrz.de`) node." + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "d0f5ede4-feb0-4896-9e65-e100cbfadee4", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "application/vnd.jupyter.widget-view+json": { + "model_id": "7eef94748b8743ed8a8dc1fae67b7926", + "version_major": 2, + "version_minor": 0 + }, + "text/plain": [ + " Searching indices: 0%| |0/1 [ ?index/s]" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
mip_erasource_idtable_idgrid_labelactivity_drsexperiment_idversionmember_idvariable_idinstitution_idprojectid
0CMIP6CESM2-WACCM-FV2AmongnCMIPhistorical20191120r1i1p1f1tasNCARCMIP6[CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1...
1CMIP6GISS-E2-1-GAmongnCMIPhistorical20180827r1i1p1f1tasNASA-GISSCMIP6[CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r...
2CMIP6CESM2-FV2AmongnCMIPhistorical20191120r1i1p1f1tasNCARCMIP6[CMIP6.CMIP.NCAR.CESM2-FV2.historical.r1i1p1f1...
3CMIP6CESM2AmongnCMIPhistorical20190308r1i1p1f1tasNCARCMIP6[CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amo...
4CMIP6GISS-E2-1-HAmongnCMIPhistorical20190403r1i1p1f1tasNASA-GISSCMIP6[CMIP6.CMIP.NASA-GISS.GISS-E2-1-H.historical.r...
5CMIP6CESM2-WACCMAmongnCMIPhistorical20190227r1i1p1f1tasNCARCMIP6[CMIP6.CMIP.NCAR.CESM2-WACCM.historical.r1i1p1...
6CMIP6MIROC6AmongnCMIPhistorical20181212r1i1p1f1tasMIROCCMIP6[CMIP6.CMIP.MIROC.MIROC6.historical.r1i1p1f1.A...
7CMIP6CMCC-CM2-SR5AmongnCMIPhistorical20200616r1i1p1f1tasCMCCCMIP6[CMIP6.CMIP.CMCC.CMCC-CM2-SR5.historical.r1i1p...
8CMIP6CMCC-CM2-HR4AmongnCMIPhistorical20200904r1i1p1f1tasCMCCCMIP6[CMIP6.CMIP.CMCC.CMCC-CM2-HR4.historical.r1i1p...
11CMIP6CMCC-ESM2AmongnCMIPhistorical20210114r1i1p1f1tasCMCCCMIP6[CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1...
\n", + "
" + ], + "text/plain": [ + " mip_era source_id table_id grid_label activity_drs experiment_id \\\n", + "0 CMIP6 CESM2-WACCM-FV2 Amon gn CMIP historical \n", + "1 CMIP6 GISS-E2-1-G Amon gn CMIP historical \n", + "2 CMIP6 CESM2-FV2 Amon gn CMIP historical \n", + "3 CMIP6 CESM2 Amon gn CMIP historical \n", + "4 CMIP6 GISS-E2-1-H Amon gn CMIP historical \n", + "5 CMIP6 CESM2-WACCM Amon gn CMIP historical \n", + "6 CMIP6 MIROC6 Amon gn CMIP historical \n", + "7 CMIP6 CMCC-CM2-SR5 Amon gn CMIP historical \n", + "8 CMIP6 CMCC-CM2-HR4 Amon gn CMIP historical \n", + "11 CMIP6 CMCC-ESM2 Amon gn CMIP historical \n", + "\n", + " version member_id variable_id institution_id project \\\n", + "0 20191120 r1i1p1f1 tas NCAR CMIP6 \n", + "1 20180827 r1i1p1f1 tas NASA-GISS CMIP6 \n", + "2 20191120 r1i1p1f1 tas NCAR CMIP6 \n", + "3 20190308 r1i1p1f1 tas NCAR CMIP6 \n", + "4 20190403 r1i1p1f1 tas NASA-GISS CMIP6 \n", + "5 20190227 r1i1p1f1 tas NCAR CMIP6 \n", + "6 20181212 r1i1p1f1 tas MIROC CMIP6 \n", + "7 20200616 r1i1p1f1 tas CMCC CMIP6 \n", + "8 20200904 r1i1p1f1 tas CMCC CMIP6 \n", + "11 20210114 r1i1p1f1 tas CMCC CMIP6 \n", + "\n", + " id \n", + "0 [CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1... \n", + "1 [CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r... \n", + "2 [CMIP6.CMIP.NCAR.CESM2-FV2.historical.r1i1p1f1... \n", + "3 [CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amo... \n", + "4 [CMIP6.CMIP.NASA-GISS.GISS-E2-1-H.historical.r... \n", + "5 [CMIP6.CMIP.NCAR.CESM2-WACCM.historical.r1i1p1... \n", + "6 [CMIP6.CMIP.MIROC.MIROC6.historical.r1i1p1f1.A... \n", + "7 [CMIP6.CMIP.CMCC.CMCC-CM2-SR5.historical.r1i1p... \n", + "8 [CMIP6.CMIP.CMCC.CMCC-CM2-HR4.historical.r1i1p... \n", + "11 [CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1... " + ] + }, + "execution_count": 31, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "intake_esgf.conf.set(indices={\"anl-dev\":False,\n", + " \"ornl-dev\":False,\n", + " \"esgf-data.dkrz.de\":True,\n", + " }\n", + " )\n", + "\n", + "cat = ESGFCatalog()\n", + "\n", + "cat = ESGFCatalog()\n", + "cat.search(\n", + " activity_id='CMIP',\n", + " experiment_id=[\"historical\",],\n", + " variable_id=[\"tas\"],\n", + " member_id='r1i1p1f1',\n", + " grid_label='gn',\n", + " table_id=\"Amon\",\n", + " institution_id=[\"MIROC\", \"NCAR\", \"NASA-GISS\", \"CMCC\"]\n", + " )\n", + "cat.df" + ] + }, + { + "cell_type": "markdown", + "id": "b00d07ff-58a7-4141-922c-9f7a97772f1e", + "metadata": { + "tags": [] + }, + "source": [ + "## Extract the Dataset ID and Pass to Rooki\n", + "Now that we have set of datasets, we need to extract the `dataset_id`, which is the unique identifier for the dataset. We can pull this from the `id` column from `intake-esgf`" + ] + }, + { + "cell_type": "markdown", + "id": "6c8ee46c-06e3-4a25-ad5d-23c7e10aa9ff", + "metadata": {}, + "source": [ + "### Separate the Dataset ID" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "e44f6c32-e4b9-4767-88eb-8099bee603e8", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120|esgf3.dkrz.de']" + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "cat.df.id.values[0]" + ] + }, + { + "cell_type": "markdown", + "id": "85d44542-5ca7-4884-b77b-aa7a469efd34", + "metadata": {}, + "source": [ + "Notice how the node information is added onto end of the file id. We need to \"chop off\" that last bit, leaving everything before the `|` character. We put this into a function to make it easier to generalize and apply." + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "363d5d3b-3da5-45a2-9981-e66969fbb227", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120'" + ] + }, + "execution_count": 37, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def separate_dataset_id(full_dataset):\n", + " return full_dataset[0].split(\"|\")[0]\n", + "\n", + "separate_dataset_id(cat.df.id.values[0])" + ] + }, + { + "cell_type": "markdown", + "id": "6e443d8c-f1d7-4f92-a0ab-ed0bbc588ced", + "metadata": {}, + "source": [ + "Now, we can apply this to the entire list within our dataframe using the following" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "044b088b-a9e0-48e9-ab93-a5a107341367", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "['CMIP6.CMIP.NCAR.CESM2-WACCM-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120',\n", + " 'CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r1i1p1f1.Amon.tas.gn.v20180827',\n", + " 'CMIP6.CMIP.NCAR.CESM2-FV2.historical.r1i1p1f1.Amon.tas.gn.v20191120',\n", + " 'CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amon.tas.gn.v20190308',\n", + " 'CMIP6.CMIP.NASA-GISS.GISS-E2-1-H.historical.r1i1p1f1.Amon.tas.gn.v20190403',\n", + " 'CMIP6.CMIP.NCAR.CESM2-WACCM.historical.r1i1p1f1.Amon.tas.gn.v20190227',\n", + " 'CMIP6.CMIP.MIROC.MIROC6.historical.r1i1p1f1.Amon.tas.gn.v20181212',\n", + " 'CMIP6.CMIP.CMCC.CMCC-CM2-SR5.historical.r1i1p1f1.Amon.tas.gn.v20200616',\n", + " 'CMIP6.CMIP.CMCC.CMCC-CM2-HR4.historical.r1i1p1f1.Amon.tas.gn.v20200904',\n", + " 'CMIP6.CMIP.CMCC.CMCC-ESM2.historical.r1i1p1f1.Amon.tas.gn.v20210114']" + ] + }, + "execution_count": 39, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dsets = [separate_dataset_id(dataset) for dataset in list(cat.df.id.values)]\n", + "dsets" + ] + }, + { + "cell_type": "markdown", + "id": "4a470bea-581c-4796-a716-a02fd02e1531", + "metadata": {}, + "source": [ + "### Compute with Rooki\n", + "Now that we have a list of IDs to pass to rooki, let's compute!\n", + "\n", + "In this case, we are:\n", + "- Subsetting from the year 1900 to 2000\n", + "- Subsetting near India using the bounds `65,0,100,35`\n", + "- Computing the yealy average\n", + "\n", + "We then check to make sure the response is okay, and if it is, return that to the user!" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "38704069-b1ad-474f-99f6-3273554df831", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "def compute_annual_mean_subset(dset_id):\n", + " # Subset by area then time\n", + " wf = ops.AverageByTime(\n", + " ops.Subset(\n", + " ops.Input(\n", + " 'tas', [dsets[0]]\n", + " ),\n", + " time='1900-01-01/2000-12-31',\n", + " area='65,0,100,35',\n", + " ),\n", + " freq=\"year\"\n", + " )\n", + " \n", + " resp = wf.orchestrate()\n", + " \n", + " if resp.ok:\n", + " ds = resp.datasets()[0]\n", + " else:\n", + " ds = xr.Dataset()\n", + " return ds" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "94609894-705d-4fdc-a235-b2f1f5747305", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_gze8fnoi/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "\n", + "
<xarray.Dataset> Size: 165kB\n",
+       "Dimensions:    (time: 101, lat: 18, lon: 15, nbnd: 2)\n",
+       "Coordinates:\n",
+       "  * lat        (lat) float64 144B 0.9474 2.842 4.737 6.632 ... 29.37 31.26 33.16\n",
+       "  * lon        (lon) float64 120B 65.0 67.5 70.0 72.5 ... 92.5 95.0 97.5 100.0\n",
+       "  * time       (time) object 808B 1900-01-01 00:00:00 ... 2000-01-01 00:00:00\n",
+       "Dimensions without coordinates: nbnd\n",
+       "Data variables:\n",
+       "    tas        (time, lat, lon) float32 109kB ...\n",
+       "    lat_bnds   (time, lat, nbnd) float64 29kB ...\n",
+       "    lon_bnds   (time, lon, nbnd) float64 24kB ...\n",
+       "    time_bnds  (time, nbnd) object 2kB ...\n",
+       "Attributes: (12/45)\n",
+       "    Conventions:            CF-1.7 CMIP-6.2\n",
+       "    activity_id:            CMIP\n",
+       "    branch_method:          standard\n",
+       "    branch_time_in_child:   674885.0\n",
+       "    branch_time_in_parent:  10950.0\n",
+       "    case_id:                1562\n",
+       "    ...                     ...\n",
+       "    sub_experiment_id:      none\n",
+       "    table_id:               Amon\n",
+       "    tracking_id:            hdl:21.14100/2ebbfd9d-97bf-4858-b893-80d31ffe8cc7\n",
+       "    variable_id:            tas\n",
+       "    variant_info:           CMIP6 CESM2 historical ensemble with WACCM6-FV2 (...\n",
+       "    variant_label:          r1i1p1f1
" + ], + "text/plain": [ + " Size: 165kB\n", + "Dimensions: (time: 101, lat: 18, lon: 15, nbnd: 2)\n", + "Coordinates:\n", + " * lat (lat) float64 144B 0.9474 2.842 4.737 6.632 ... 29.37 31.26 33.16\n", + " * lon (lon) float64 120B 65.0 67.5 70.0 72.5 ... 92.5 95.0 97.5 100.0\n", + " * time (time) object 808B 1900-01-01 00:00:00 ... 2000-01-01 00:00:00\n", + "Dimensions without coordinates: nbnd\n", + "Data variables:\n", + " tas (time, lat, lon) float32 109kB ...\n", + " lat_bnds (time, lat, nbnd) float64 29kB ...\n", + " lon_bnds (time, lon, nbnd) float64 24kB ...\n", + " time_bnds (time, nbnd) object 2kB ...\n", + "Attributes: (12/45)\n", + " Conventions: CF-1.7 CMIP-6.2\n", + " activity_id: CMIP\n", + " branch_method: standard\n", + " branch_time_in_child: 674885.0\n", + " branch_time_in_parent: 10950.0\n", + " case_id: 1562\n", + " ... ...\n", + " sub_experiment_id: none\n", + " table_id: Amon\n", + " tracking_id: hdl:21.14100/2ebbfd9d-97bf-4858-b893-80d31ffe8cc7\n", + " variable_id: tas\n", + " variant_info: CMIP6 CESM2 historical ensemble with WACCM6-FV2 (...\n", + " variant_label: r1i1p1f1" + ] + }, + "execution_count": 42, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "compute_annual_mean_subset(dsets[0])" + ] + }, + { + "cell_type": "markdown", + "id": "4ecacf51-e35f-4810-8250-3e90a1b5888a", + "metadata": {}, + "source": [ + "Now that it works with a single dataset, let's do this for all the datasets and put them into a dictionary with the dataset ids as the keys." + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "a52e7d71-18fd-4bbe-91a7-fa86725ad4de", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_shm9lihm/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_knwmboqr/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_hwnk7sbm/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_j_w92ac9/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_0qmd9id3/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_53g0pvrs/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_4p48gwjo/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_5p_02p66/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_0d20f3r4/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n", + "Downloading to /var/folders/bw/c9j8z20x45s2y20vv6528qjc0000gq/T/metalink_xncn6l3t/tas_Amon_CESM2-WACCM-FV2_historical_r1i1p1f1_gn_19000101-20000101_avg-year.nc.\n" + ] + } + ], + "source": [ + "dset_dict = {}\n", + "for dset in dsets:\n", + " dset_dict[dset] = compute_annual_mean_subset(dset)" + ] + }, + { + "cell_type": "code", + "execution_count": 81, + "id": "50cecbf0-9a45-4565-a559-cb23f0f8da94", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": { + "text/plain": [ + "'CMIP6.CMIP.NASA-GISS.GISS-E2-1-G.historical.r1i1p1f1.Amon.tas.gn.v20180827'" + ] + }, + "execution_count": 81, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "list(dset_dict.keys())[1]" + ] + }, + { + "cell_type": "markdown", + "id": "4a77cda9-536f-4fc2-8e89-9ce2e4937810", + "metadata": {}, + "source": [ + "## Visualize the Output\n", + "Let's use hvPlot to visualize. The datasets are stored in a dictionary of datasets, we need to:\n", + "- Extract a single key\n", + "- Plot a contour filled visualization, with some geographic features" + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "id": "1b08b0a0-7add-4a50-8e62-9bb88b98cc7c", + "metadata": { + "tags": [] + }, + "outputs": [ + { + "data": {}, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": {}, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "application/vnd.holoviews_exec.v0+json": "", + "text/html": [ + "
\n", + "
\n", + "
\n", + "" + ], + "text/plain": [ + "Column\n", + " [0] HoloViews(DynamicMap, sizing_mode='fixed', widget_location='bottom')\n", + " [1] WidgetBox(align=('center', 'end'))\n", + " [0] DiscreteSlider(name='time', options={'1900-01-01 00:00:00': cf...}, value=cftime.DatetimeNoLeap(1900...)" + ] + }, + "execution_count": 102, + "metadata": { + "application/vnd.holoviews_exec.v0+json": { + "id": "p25555" + } + }, + "output_type": "execute_result" + } + ], + "source": [ + "dset_dict['CMIP6.CMIP.NCAR.CESM2.historical.r1i1p1f1.Amon.tas.gn.v20190308'].tas.hvplot.contourf(x='lon',\n", + " y='lat',\n", + " cmap='Reds',\n", + " levels=20,\n", + " clim=(250, 320),\n", + " features=[\"land\", \"ocean\"],\n", + " alpha=0.7,\n", + " widget_location='bottom',\n", + " clabel=\"Yearly Average Temperature (K)\",\n", + " geo=True)" + ] + }, + { + "cell_type": "markdown", + "id": "33c9fce3-722a-4cc9-8b4c-5d2038cc981f", + "metadata": {}, + "source": [ + "---" + ] + }, + { + "cell_type": "markdown", + "id": "86e3979e-505a-455c-8753-6cd5c7f0b219", + "metadata": {}, + "source": [ + "## Summary\n", + "Within this notebook, we learned how to specify a specific index node to search from, pass discovered datasets to rooki, and chain remote-compute with several operations using rooki. We then visualized the output using hvPlot, leading to an interactive plot!\n", + "\n", + "### What's next?\n", + "More adaptations of the intake-esgf + rooki to remotely compute on ESGF data." + ] + }, + { + "cell_type": "markdown", + "id": "bbda4631-017e-4295-b2a5-e97fe5197cf8", + "metadata": {}, + "source": [ + "## Resources and references\n", + " - [intake-esgf documentation](https://intake-esgf.readthedocs.io/en/latest/)\n", + " - [rooki documentation](https://rooki.readthedocs.io/en/latest/)\n", + " - [Working with geographic data with hvPlot](https://hvplot.holoviz.org/user_guide/Geographic_Data.html)\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c36f52a0-c7c3-4e33-a35f-054b006fe63c", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.10.14" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}