Skip to content

Documentation

Jump to bottom
devRideg edited this page Jan 14, 2021 · 16 revisions

Introduction

This project was done in order to evaluate the feasability of using interpolated weather data in replacement of actual weather observation stations. The data set used consisted of hourly observations and interpolations collected from LantMet and SMHI respectively. The user should have access to the MESAN data set MESAN_CSV.tar.gz. Otherwise the user may use the GRIB2CSV.ipynb notebook to fetch, and filter GRIB-data. Note that this is very slow. The data should be arranged according to the format specified in file_structure_example.png

How to execute the notebooks in Galaxy climate interactive JupyterLab

  • Log in to Galaxy/climate at https://climate.usegalaxy.eu/
  • In the tool search bar, search for "Interactive Climate Notebook"
  • Execute a new notebook
  • In the panel menu choose user/Active InteractiveTools
  • Choose "Climate Interactive Tool", this should open your new notebook
  • Upload all files from the GitHub repository into the notebook directory
  • Choose one of the notebooks to run

Getting started.

If you have MESAN_CSV.tar.gz

  • Create a new instance of an interactive climate notebook in Galaxy.
  • Upload and unpack the .tar -file by the following terminal command: "tar -xzvf MESAN_CSV.tar.gz"
  • Upload METCOMP_utils.py. This script holds some I/O functionality used by most scripts.
  • Fetch and save LantMet data. This can be done by running "save_LANTMET_dataset.ipynb". Should the LantMet API fail to respond, repeat the stations which failed.
  • Now, the datasets should be available and you are ready to upload the notebooks/scripts you want to run.

If you do not have MESAN_CSV.tar.gz

  • Create a new instance of an interactive climate notebook in Galaxy.
  • Upload the GRIB2CSV.ipynb notebook.
  • Specify time interval and a list of coordinates. Each coordinate is specified in the following format: {'id': 'NAME', 'lat': 55.5555, 'lon': 13.3333}
  • Upload METCOMP_utils.py. This script holds some I/O functionality used by most scripts.
  • Fetch and save LantMet data. This can be done by running "save_LANTMET_dataset.ipynb". Should the LantMet API fail to respond, repeat the stations which failed.
  • Now, the datasets should be available and you are ready to upload the notebooks/scripts you want to run.

Code descriptions

METCOMP_utils.py Function collection containing functions expected to be used in several scripts. Needs to be in the same directory as the script/notebook that is executed.

Sampling

Examine_MESAN_files.py Checks that all stations have the same amount of timestamps, if the logged parameters are the same between corrdinates and if the utf-8 encoding might have failed.

MESAN_Recording.py Samples the 24h MESAN API for every LantMet stations and saves the fetched information in a .txt -file with a JSON representation of the information. This was scheduled to run twice per day.

METCOMP_utils.py Collection of useful functionalities across scripts. Usually imported by "from METCOMP_utils import *"

Test_MESAN_RECORDING.py Checks for timestamps in sampled data. Used to debug MESAN_Recording during development.

GRIB

GRIB2CSV.ipynb Fetch archived GRIB-files and extract tabular data. Note: This may take several days if the interval/list of stations is large.

Requirements:

  • METCOMP_utils.py
  • Executed on Galaxy climate interactive JupyterLab

Demo.ipynb Simple demonstration how to read extracted tabular data and fetch historical LANTMET station data. Both these functions can be found in METCOMP_utils.py. Simple plot example at the end.

Requirements:

  • Executed GRIB2CSV example. (Also specified in Demo.ipynb)

Visualizations

AvarageDailyErrors.ipynb Plots diurnal absolute errors for all stations.

Analysis_Tools.ipynb General statistical visualizations.

Boxplots.ipynb Creates boxplots for all seasons given a station and parameter.

CorrelationTable.ipynb Creates table with Pearson's correlation coefficients for all stations.

MeanErrorSTDTable.ipynb Creates table with mean and standard deviation of absolute error for each station.

ScatterPlots.ipynb Grid of station specific scatter plots where each row and column corresponds to a parameter and season (For all stations). A linear model is fitted to the data.

Save_LANTMET_dataset.ipynb Small code snippet to fascilitate local storage of LANTMET weather data from their API.

ParameterPlots.ipynb Some visualisation examples. Correlation plots for LANTMET/MESAN parameters with confidence ellipse for each month. Error distribution histograms/boxplots for each month.

Requirements:

  • METCOMP_utils.py
  • Extracted MESAN .csv -files using GRIB2CSV. (Optional time interval)

correlation.ipynb Creates scatter plot for all parameters and data points.

Clone this wiki locally