Helene BEST track perturbation failed

[FariborzDaneshvar-NOAA]

Unlike the OFCL track, prep.sbatch for the BEST track of Hurricane Helene, 2024 failed with this message:

+ setup_ensemble --track-file /work2/noaa/nos-surge/shared/nhc_hurricanes_test_wwm/helene_2024_7cbea927-ed08-4442-bbf9-15139e783eb5_GAHM_regression_wwm_korobov_19_BEST/nhc_track/hurricane-track.dat --output-directory /work2/noaa/nos-surge/shared/nhc_hurricanes_test_wwm/helene_2024_7cbea927-ed08-4442-bbf9-15139e783eb5_GAHM_regression_wwm_korobov_19_BEST/setup/ensemble.dir/ --num-perturbations 19 --mesh-directory /work2/noaa/nos-surge/shared/nhc_hurricanes_test_wwm/helene_2024_7cbea927-ed08-4442-bbf9-15139e783eb5_GAHM_regression_wwm_korobov_19_BEST/mesh/ --hires-region /work2/noaa/nos-surge/shared/nhc_hurricanes_test_wwm/helene_2024_7cbea927-ed08-4442-bbf9-15139e783eb5_GAHM_regression_wwm_korobov_19_BEST/mesh/hires --sample-from-distribution --sample-rule korobov --date-range-file /work2/noaa/nos-surge/shared/nhc_hurricanes_test_wwm/helene_2024_7cbea927-ed08-4442-bbf9-15139e783eb5_GAHM_regression_wwm_korobov_19_BEST/setup/dates.csv --nwm-file /work2/noaa/nos-surge/smani/data/nwm/NWM_v2.0_channel_hydrofabric/nwm_v2_0_hydrofabric.gdb --tpxo-dir /work2/noaa/nos-surge/smani/data/tpxo --variables cross_track along_track radius_of_maximum_winds_persistent max_sustained_wind_speed --use-wwm --perturb-features isotach_adjustment --pahm-model gahm helene 2024
/work2/noaa/nos-surge/daneshva/miniconda3/envs/stormworkflow/lib/python3.11/site-packages/pyschism/forcing/hycom/gofs.py:8: UserWarning: The seawater library is deprecated! Please use gsw instead.
  import seawater as sw
can't find the 'adcircpy' module
can't find the 'adcircpy' module
can't find the 'adcircpy' module
/work2/noaa/nos-surge/daneshva/miniconda3/envs/stormworkflow/lib/python3.11/site-packages/stormevents/nhc/track.py:183: UserWarning: It is recommended to specify the file_deck and/or advisories when reading from file
  warnings.warn(
slurmstepd: error: *** JOB 4740395 ON hercules-07-39 CANCELLED AT 2025-03-10T17:20:22 DUE TO TIME LIMIT ***

[FariborzDaneshvar-NOAA]

I changed the RMW perturbation variable and fill method from regression to persistent, but it didn't help. Still getting the same error.

[FariborzDaneshvar-NOAA]

I also ran test_perturb.py for Helene BEST track:

('helene', 2024, datetime(2024, 9, 25, 6), datetime(2024, 10, 1, 12))

And the test passed!

[SorooshMani-NOAA]

What files are already created under ensemble.dir after this failure?

[FariborzDaneshvar-NOAA]

What files are already created under ensemble.dir after this failure?

@SorooshMani-NOAA these 3 files are created under ensemble.dir:

• track_to_perturb.dat
• track_files/original.22
• track_files/original.json

[SorooshMani-NOAA]

So it is at

ondemand-storm-workflow/stormworkflow/prep/setup_ensemble.py

Line 202 in 20bc2aa

perturbations = perturb_tracks(

that is stops. Can you put a breakpoint before this call and try to see what all the args to the function are? Maybe we pass something that breaks it. Then if we find out we can add it as a test to ensembleperturbation.