Issues with xr_concat_sameRun for GFS only

[v7nguyen]

When running xr_concat_sameRun for different weather sources, all of the other sources work correctly except for GFS. When running the function, it will only download and concatenate the first 2 grib files and then not the rest, even though those grib files exist.
Has anyone else encountered this behavior before? Thanks in advance!

[blaylockbk]

Hi @v7nguyen, I found a bug that prevented the downloaded files from having unique file names! This is fixed in the master branch (see de70cb7).

Also, I have a new method for concatenating multiple xarray Datasets together you might like to try. It is a little faster than xr_concat_sameRun

from herbie.tools import fast_Herbie_xarray

ds = fast_Herbie_xarray(
    searchString=":(?:TMP|RH|APTMP):2 m above ground", 
    DATES='2022-04-01',
    fxx=range(7),
    model='gfs', 
    product='pgrb2.0p25',
)
ds

[v7nguyen]

I recently began having issues with my vscode kernel dying when executing fast_Herbie_xarray. It dies about 90% of the time. I have tried changing the number of threads to fewer which might have helped but I am not sure. Do you have any advice/experience with this?

[blaylockbk]

Hmm, I wonder if this is a memory issue. How much data are you trying to load? Does your kernel still die when you try loading only a few? All threads share the same memory, so that could be the issue.

[v7nguyen]

So I've finally done root cause analysis and it seems that the error only happens when I am pulling wind data from HRRR and am using more than a single thread. When using a single thread it works for wind data, and when pulling ambient data (TMP|RH|DPT) I can use as many threads as I would like. Attached is a code snippet. Let me know if you're able to replicate this error or if it's an issue on my local. Thanks in advance!

[v7nguyen]

This code results in the kernel dying after FYI!

[blaylockbk]

I wasn't able to reproduce the error running on Linux with Python 3.9.7
Can you tell me more about your setup? What type of computer are you on and the python version.

---

from herbie.tools import fast_Herbie_xarray

ds = fast_Herbie_xarray(
    searchString=":(U|V)GRD:10 m",
    DATES="2022-05-05 12:00",
    fxx=range(0,3,1),
    model='hrrr',
    product='sfc',
    verbose=True,
    max_threads=2
)

I also could read all F00-F18 fields with no issue

ds = fast_Herbie_xarray(
    searchString="(?:U|V)GRD:10 m",
    DATES="2022-05-05 12:00",
    fxx=range(18),
    model='hrrr',
    product='sfc',
    verbose=True,
    #max_threads=2
)
ds

[blaylockbk]

And thanks @v7nguyen for finding/reporting this bug 😊

[v7nguyen]

Thank you for your quick response!!! Much appreciated :)

[v7nguyen]

You're doing great work, trying to be more like you haha