Using Herbie in a Colab notebook

[purpal9]

Hello Brian, long time since the UVA and NASA internships! And everyone else who might help!

I am using Herbie in a google colab notebook as part of a lesson on training a random forest model to predict next-hour fire radiative power. I thought using Herbie to pull the RAP files from the mighty web would be a good idea. My understanding is that Herbie bypasses the more classes "xr.open_Dataset pynio" commands for grib2. So my question is that before, I pulled RAP files and stored them in a directory, and then opened the grib2 with pynio, and extracted the variables I needed and was further able to only extract the grid cell info I needed with a closest point formula from the fire lat/lon. But if I use Herbie, how do I treat the Herbie RAP file like what I'm used to so that I can save a variable in an array? For instance, it used to be something like "temp = grbs['TMP_P0_L103_GRLL0'].values.flatten()" to make an array. The "ds" command in the tutorial makes a dataset.. I just need the values to be honest and none of the meta data. Thanks - Christina

[blaylockbk]

Hi Christina! It has been a while. Good to hear from you.

Sounds interesting what you are doing. I don't have any experience using colab and haven't used pynio in a very long time.

Let's see how I do at answering your questions and giving some ideas...

---

If you like using pynio, you could just use Herbie to download the files you want and then use pynio to read those downloaded file.

from herbie import Herbie

H = Herbie('2024-01-01', model='rap')
my_file = H.download()

# The value of `my_file` is PosixPath('/home/blaylock/data/rap/20240101/rap.t00z.awp130pgrbf00.grib2')

# Use pynio to read/process that file

When you download a Herbie object, it returns the path to the downloaded file. You could use that file with pynio however you are familiar. If the file has already been downloaded, Herbie won't redownload unless you tell it to.

---

Going the xarray route, Herbie uses cfgrib to read the GRIB2 file into a DataFrame. You can select a specific variable/level you want using the searchString argument (you can use H.inventory() to look at the contents of the GRIB file).

From an xarray dataset, it's easy to convert the dataset data into numpy arrays/vectors easily so you don't have to deal with extra metadata.

H = Herbie("2024-01-01", model="rap")

# Get just 2-m temperature data as a dataset
ds = H.xarray(searchString="TMP:2 m")

# these are all numpy vectors
lat = ds.latitude.values.flatten()
lon = ds.longitude.values.flatten()
tmp_2m = ds.t2m.values.flatten()

---

You might be interested in Herbie's nearest neighbor xarray accessor that you can read more about here; it throws some warnings, but it does work.

---

Does any of that give you some ideas?

[purpal9]

Thanks - I think that this answered what I needed. And I recall coming across your nearest neighbor before! Very good stuff. My last quick question is, is there a way to pull files by hour? Currently it's a day. Or do we use forecast hour for 1-22 UTC? Thanks :)

[blaylockbk]

Great!
The date is the model run time, and you can specify fxx to give the forecast lead time in hours.

# Model run at 06z, valid at 06z
H = Herbie("2024-01-01 06:00", model="rap", fxx=0)

# Model run at 00z, valid at 06z
H = Herbie("2024-01-01 00:00", model="rap", fxx=6)

[purpal9]

I think I have one more question. Since we're looking to pull multiple variables to use as ndarrays, how does one do that? The example only seems to work on the one variable. Specifically, we're looking at using TMP: 2m, but also:
UGRAD:10 m
VGRAD:10 m
DPT:10 m
GUST
SOILW
VGTYP
TOSIL

And the subset with search is not entirely clear on how to do this and then store them to arrays. I tried something like the following, but it didn't work:
dss = H.xarray(searchString="UGRD:10 m")
It throws:
FileNotFoundError: [Errno 2] No such file or directory: '/root/data/rap/20240101/subset_6befe7f6__rap.t00z.awp130pgrbf00.grib2'

Perhpas there's another way? And then, you have
"tmp_2m = ds.t2m.values.flatten()"
But didn't necessarily explain where t2m comes from, since we don't define it. It seems internal and also related to the ecCODES
":2t:" | 2-m temperature -- | --
But that is missing an m.

Lastly, if I'm doing a loop by time and want to pull a second hour, is there a clearing command to set up a new Herbie object? Or will pulling and saving another file to H be enough?

[blaylockbk]

The searchString argument can take a regular experssion to match the different GRIB index labels. For example, you could use searchString="(?:TMP:2 m|GRD:10 m|DPT:2 m)" to get the wind, temp, and dew point. The best way to test different searchStrings is to look at the file inventory with a search string.

# show full inventory
H = Herbie("2024-01-01 06:00", model="rap", fxx=0)
H.inventory()

When giving a regex searchString, it will search the values in the search_this column. In the next example, I use the regex syntax (?:item1|item2|item3):

# find select fields
H = Herbie("2024-01-01 06:00", model="rap", fxx=0)
print(H.grib)
H.inventory("(?:TMP:2 m|GRD:10 m|DPT:2 m|GUST)")

It looks like the soil and veg type fields are in the native file, so you'll have to change from the default product...

H = Herbie("2024-01-01 06:00", model="rap", fxx=0, product="awp130bgrb")
print(H.grib)
H.inventory("(?:SOIL|VGTYP|TOSIL)")

Once you have a searchString that finds the variables you want, you can either download it or load it into xarray.

H.download(searchString)
H.xarray(searchString)

The xarray feature loads the data with cfgrib (which uses the eccodes nomenclature for variable abbreviations), and cfgrib doesn't merge all the data into the same dataset. For instance, if you did this...

H = Herbie("2024-01-01 06:00", model="rap", fxx=0)
ds = H.xarray("(?:TMP:2 m|GRD:10 m|DPT:2 m|GUST)")
len(ds)

you would actually get three datasets, one for the 10-m products, one for the surface products, and one for the 2-m products. Similarly,

H = Herbie("2024-01-01 06:00", model="rap", fxx=0, product="awp130bgrb")
ds = H.xarray("(?:SOIL|VGTYP|TOSIL)")
len(ds)

would return two datasets, one for the depthBelowLandLayer data and one for the surface data.

Merging these products in a single dataframe would be an exercise in xarray.

---

And the subset with search is not entirely clear on how to do this and then store them to arrays. I tried > something like the following, but it didn't work:
dss = H.xarray(searchString="UGRD:10 m")
It throws:
FileNotFoundError: [Errno 2] No such file or directory: '/root/data/rap/20240101/subset_6befe7f6__rap.t00z.awp130pgrbf00.grib2'

I'm not sure why you are getting an error here. It looks like the file was removed before it could be read.

---

And then, you have
"tmp_2m = ds.t2m.values.flatten()"
But didn't necessarily explain where t2m comes from, since we don't define it. It seems internal and also > related to the ecCODES
":2t:" | 2-m temperature -- | --
But that is missing an m.

The t2m is the xarray variable name as given by cfgrib, used to read the data. From here...

H = Herbie("2024-01-01 06:00", model="rap", fxx=0)
ds = H.xarray("(?:TMP:2 m|GRD:10 m|DPT:2 m|GUST)")
ds[1]

---

Lastly, if I'm doing a loop by time and want to pull a second hour, is there a clearing command to set up > a new Herbie object? Or will pulling and saving another file to H be enough?

You can just overwrite the H value. Something like this...

import pandas as pd

for date in pd.date_range(start="2024-1-1", periods=3, freq="1h"):
    H = Herbie(date, model="rap", fxx=0, verbose=False)
    ds = H.xarray("(?:TMP:2 m|DPT:2 m)")
    print(
        f"{date}, the maximum tmp={ds.t2m.max().item():.2f}, the maximum dpt={ds.d2m.max().item():.2f}"
    )