Challenge 13 - Bridging the Gap: Reconciliation of Model Data with Observations

[DiegoDomenici]

Stream 1 - Data Visualization

Mentors

• Iryna Rozum
• James Varndell

Skill Required

• Python
• Frontend web development (React is desirable)
• Experience or interest in WebGL or alternative technologies is desirable
• Experience or interest in processing geospatial data (Python xarray) is desirable

Goal

Develop an interactive web application to study consistency between model data and observations over long periods of time, using frontend technologies like WebGL for high-performance computations and responsiveness.

---

Note: The funding source for this challenge is ECMWF funding. For details on the eligibility, please refer to Article 3 of the Terms and Conditions.

---

Description of the Challenge

Copernicus Climate Data Store is a cloud-based repository providing access to diverse types of data, such as Essential Climate Variables (ECVs), reanalysis products, satellite observations, and other datasets.
ERA5 reanalysis is the fifth generation ECMWF reanalysis for the global climate and weather for the past 8 decades. It combines model data with observations into a globally complete and consistent dataset.
Comparing reanalysis data with satellite observations is necessary because both data sources have inherent limitations, and combining them provides a more accurate, reliable, and comprehensive understanding of the Earth's climate and weather systems. While reanalysis offers a continuous, "map without gaps" representation by blending models with data, it relies on complex, potentially biased, model simulations. Conversely, satellites offer raw, real-time, high-resolution observations of the Earth, but these are often limited by cloud cover, specific viewing angles, and temporal gaps.

The aim of this project is to explore the possibility to use WebGL to compute time series of climatological anomalies and to create interactive visualisations in the browser.

The work in this project can be split into two parts:

1. Data preparation. First, selected variables from the ERA5 and satellite datasets need to be programmatically retrieved from the Climate Data Store and pre-processed to produce area-averaged absolute values and climatological anomalies over IPCC AR6 regions, eg. monthly mean, quarterly, decadal, etc. Ideally, the data processing should be part of the “on-the-fly” workflow using WebGL or similar technologies.
2. Data visualization.
   a. App layout – The app should provide a drop down selection to choose a variable and a clickable map to display and select AR6 regions.
   b. Visualisation – Several interactive charts displaying absolute values and climatological anomalies for the selected AR6 regions. All charts should be aligned with respect to start and end times for the time axis. Legends should be interactive to toggle between plotted datasets.

The main challenge will be to process high-resolution satellite data on-the-fly and to render data efficiently with a sufficiently fast response time for the application to be interactive. The complexity of the development can be tailored to the interests of the applicants.

Evaluation criteria

• Innovative approach
• Feasibility
• Easy to maintain / Future-proof approach
• Transferability

[Lizzy-LE]

Dear Mentors,

I 'm Lizzy and I am interested in applying, however I would have some technical questions I'd like to ask. Let me quote the original text, followed by my question(s), hoping you could you be so kind to confirm if (or not) I am understanding this correctly. And if not, could you be so kind to elaborate? Thx.

---

The aim of this project is to explore the possibility to use WebGL to compute time series of climatological anomalies and to create interactive visualisations in the browser.

It is my understanding that instead of the classic tools, such as xarray+dask, heavy lifting and computation should be done in javascript, using WEBGL? Based on my past experience and limited knowledge, the data would be downloaded (API) separately, processed and, when ready, uploaded to the front end. Here, instead, it should be done all "on-the-fly", just in time or quasi-real-time, on the front end? Am I understanding correctly?

The work in this project can be split into two parts:

1. Data preparation. First, selected variables from the ERA5 and satellite datasets need to be programmatically retrieved from the Climate Data Store and pre-processed to produce area-averaged absolute values and climatological anomalies over IPCC AR6 regions, eg. monthly mean, quarterly, decadal, etc. Ideally, the data processing should be part of the “on-the-fly” workflow using WebGL or similar technologies.

Usually I do the programmatic data download and pre-processing in Python - and this is common business. Instead, I think to understand that in this case, the actual computation and heavy lifting (grouping, slicing, aggregating, reshaping) should be done in WEBGL? To my present knowledge and skill, the API retrieval part could be feasible, but the actual processing "on the front-end" (WEBGL) is unexplored virgin territory to me. Is this the whole part of the challenge? Persuading WEBGL to do what xarray does and is good at? And also: why call it "Data Preparation" if no data is actually prepared, but almost thrown directly at the frontend?

2. Data visualization.
   a. App layout – The app should provide a drop down selection to choose a variable and a clickable map to display and select AR6 regions.
   b. Visualisation – Several interactive charts displaying absolute values and climatological anomalies for the selected AR6 regions. All charts should be aligned with respect to start and end times for the time axis. Legends should be interactive to toggle between plotted datasets.

This sounds feasible to me. For other projects, I have used django in the past to build webpages and have control over all the moving parts and the process governing them. Is django acceptable?

The main challenge will be to process high-resolution satellite data on-the-fly and to render data efficiently with a sufficiently fast response time for the application to be interactive. The complexity of the development can be tailored to the interests of the applicants.

And here comes my other question: how feasible would it be to have the downloading + processing on a "separate track"? I am thinking in something like asynchronous tasks to download, process and upload the results. I have experience in doing this, but I am afraid it would clash with the statement of the project goal: "with a sufficiently fast response time for the application to be interactive".

All that said, I would love to hear your comments and thoughts.
And of course THANK YOU very much for the time you will dedicate to answer my questions.
Kind regards,
Lizzy

[Lizzy-LE]

Hi again, just a brief follow-up.

This morning I started messing around with this idea, and using django5 I could recreate the layout and functional structure of maps + domains/regions. When selecting a region, plots (so far only plot titles) are updated dynamically.

Now, as soon as I will understand the mechanics of how anomalies are calcualated/precomputed/prepared (see my previous message), I could move on the the next phase.

Just to point out that the "vessel" is ready, I only need to understand how to build the "engines"....

[IrynaRozum]

Hello Lizzy,

Thank you for your interest in this challenge. We will reply to your questions next week.

best regards,
Iryna

[IrynaRozum]

Hello Lizzy,

In this project we would like to explore if it is possible to repurpose WebGL for processing high-resolution data like satellite-observations. While WebGL is designed for rendering, online resources show that it can be repurposed for quick data processing by utilizing the GPU for parallel computing.
Satellite and model data are gridded, so it should be possible to load this data into WebGL textures (converting to binary or image formats first), then use shader programs (GLSL) to perform the subtraction (Value - Climatology) for thousands of pixels simultaneously, and output the resulting anomalies. Note that reference climatologies will be pre-calculated. Yes, "data preparation" part is probably the wrong word, but we wanted to keep the options open if some other very quick way of processing data can be developed (the standard python xarray can be a bit slow here).

React would be preferred for the frontend development, and should be easily integrated with our other apps. James will provide more information here.

best regards,
Iryna

[Lizzy-LE]

Hi @IrynaRozum and @JamesVarndell,

Thanks for the feedback. I’m exploring the idea you illustrated in your previous message.

I’m splitting a large spatio-temporal data cube into smaller time chunks called slabs, fetching and processing each slab progressively, and gradually populating the dataset in memory. At the same time, I apply hierarchical reduction, downsampling each slab’s spatial grid step by step (e.g., 256→128→64…) by averaging small blocks. This allows me to handle and analyze large datasets efficiently without overloading memory.

So far, the slabbing approach seems to work well in WebGL (still testing), but the main bottleneck appears to be bringing the full model or satellite data to the frontend. Even with precomputed slabs, the complete measured data cube with the full time dimension still needs to be read in, which slows everything down.

I have a couple of questions:

1. Are there strategies to efficiently stream or partially load such large datasets into the browser without hitting memory or transfer bottlenecks?

2. Do we have an estimate of the average size of a typical data cube? I’m assuming daily data over 80 years, so the time dimension is roughly known, but what about spatial dimensions?

Could you help me identify the best path to explore for tackling these challenges?

Thanks so much for your kind help!!!