Adds GDAS GEMPAK jobs to rocoto task mesh #2073

HenryRWinterbottom · 2023-11-16T18:03:50Z

Description

This PR addresses issue #1219. Rocoto task jobs/rocoto/gempak.sh was updated to support JGDAS_ATMOS_GEMPAK. Rocoto task jobs/rocoto/gempakmeta.sh was added to support JGFS_ATMOS_GEMPAK_META and JGDAS_ATMOS_GEMPAK_META_NCDC.

Resolves #1219
References #1222

Type of change

New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

The Rocoto XML configuration has been executed as follows.

user@host$: cd /scratch1/NCEPDEV/da/Henry.Winterbottom/trunk/global-workflow.develop.20231117/workflow
user@host$: ./setup_xml.py /scratch1/NCEPDEV/da/Henry.Winterbottom/work/GLOBAL_WORKFLOW/x002_gfsv17_issue_1219/x002_gfsv17_issue_1219.xml

The resulting Rocoto workflow is as follows.

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202103231200             gfsstage_ic                    51952235              QUEUED                   -         0           0.0
202103231200                 gfsfcst                           -                   -                   -         -             -
202103231200            gfspost_f000                    51952236              QUEUED                   -         0           0.0
202103231200            gfspost_f006                    51952237              QUEUED                   -         0           0.0
202103231200            gfspost_f012                    51952238              QUEUED                   -         0           0.0
202103231200                 gfsvrfy                           -                   -                   -         -             -
202103231200             gfsmetpg2g1                           -                   -                   -         -             -
202103231200             gfsmetpg2o1                           -                   -                   -         -             -
202103231200             gfsmetppcp1                           -                   -                   -         -             -
202103231200               gfsgempak                           -                   -                   -         -             -
202103231200           gfsgempakmeta                           -                   -                   -         -             -
202103231200       gfsgempakmetancdc                           -                   -                   -         -             -
202103231200    gfsgempakncdcupapgif                           -                   -                   -         -             -
202103231200      gfsgempakpgrb2spec                           -                   -                   -         -             -
202103231200                 gfsarch                           -                   -                   -         -             -
202103231200              gfscleanup                           -                   -                   -         -             -

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

HenryRWinterbottom · 2023-11-16T18:04:20Z

@WalterKolczynski-NOAA I am not sure if this is what you had in mind. Please let me know if I am misunderstanding.

jobs/rocoto/gempakmeta.sh

jobs/rocoto/gempak.sh

jobs/rocoto/gempakmeta.sh

HenryRWinterbottom · 2023-11-16T23:09:10Z

Do you mean beneath workflow/rocoto or somewhere else?

WalterKolczynski-NOAA · 2023-11-16T23:34:27Z

Do you mean beneath workflow/rocoto or somewhere else?

Yes. You need to add all the stuff for the new jobs.

HenryRWinterbottom · 2023-11-17T00:37:34Z

@WalterKolczynski-NOAA Done.

WalterKolczynski-NOAA · 2023-11-17T07:10:03Z

The CI tests won't exercise these jobs. You'll need to test them manually. The jobs themselves likely will not succeed, but make sure the jobs launch when they should.

KateFriedman-NOAA

Mostly good but need to confirm if one job runs for gdas, or gfs, or both and make adjustments based on when it should run. Left comments, although hurridly since I am wrapping up work so hopefully they make sense. If my comments have been addressed and others approve before I return from holiday leave then ignore my review and proceed. Thanks for working on this @HenryWinterbottom-NOAA !

KateFriedman-NOAA · 2023-11-17T21:04:45Z

jobs/rocoto/gempakmetancdc.sh

+export jobid="${job}.$$"
+
+# Execute the JJOB
+"${HOMEgfs}/jobs/JGDAS_ATMOS_GEMPAK_META_NCDC"


Is this only for gdas? Is there a gfs version of the job? Can't remember what happens in ops off the top of my head.

Also, if you didn't already, review the gempaks jobs in this ops ecflow def file: https://github.com/NOAA-EMC/global-workflow/blob/dev/gfs.v16/ecf/defs/gfs_v16_3.def

both:

gempak

gdas-only:

gempakmetancdc

gfs-only:

gempakmeta

gempakncdcupapgif

gempakpgrb2spec

KateFriedman-NOAA · 2023-11-17T21:05:55Z

workflow/applications/gfs_cycled.py

@@ -230,6 +230,10 @@ def get_task_names(self):

        if self.do_gempak:
            gfs_tasks += ['gempak']
+            gfs_tasks += ['gempakmeta']
+            gfs_tasks += ['gempakmetancdc']


If this is meant for only gdas then you need to add this task to the gdas_tasks above. If this is only for gfs then it's fine but see other comment. If both then add to gdas_tasks.

workflow/rocoto/gfs_tasks.py

WalterKolczynski-NOAA

Plus what Kate pointed out about making sure jobs run only for the appropriate RUN.

WalterKolczynski-NOAA · 2023-11-20T05:26:56Z

workflow/applications/applications.py

+        self.do_gempakmeta = _base.get('DO_GEMPAK', False)
+        self.do_gempakmetancdc = _base.get('DO_GEMPAK', False)
+        self.do_gempakncdcupapgif = _base.get('DO_GEMPAK', False)
+        self.do_gempakpgrb2spec = _base.get('DO_GEMPAK', False)


These are not needed. Only self.do_gempak is being used.

I added this because my Rocoto mesh wasn't picking up the jobs. Maybe it was a red herring and I will look again but it took me sometime to figure out and why and this seemed to resolve the issue.

NVM the above. It must have been a red herring.

workflow/rocoto/gfs_tasks.py

WalterKolczynski-NOAA · 2023-11-29T21:40:07Z

@HenryWinterbottom-NOAA I think this may be ready after you resolve the conflicts

HenryRWinterbottom requested a review from WalterKolczynski-NOAA November 16, 2023 18:04

github-advanced-security bot found potential problems Nov 16, 2023

View reviewed changes

jobs/rocoto/gempakmeta.sh Fixed Show fixed Hide fixed

jobs/rocoto/gempakmeta.sh Fixed Show fixed Hide fixed

WalterKolczynski-NOAA requested changes Nov 16, 2023

View reviewed changes

jobs/rocoto/gempak.sh Outdated Show resolved Hide resolved

jobs/rocoto/gempak.sh Outdated Show resolved Hide resolved

jobs/rocoto/gempakmeta.sh Outdated Show resolved Hide resolved

jobs/rocoto/gempakmeta.sh Outdated Show resolved Hide resolved

HenryRWinterbottom self-assigned this Nov 17, 2023

HenryRWinterbottom added the feature New feature or request label Nov 17, 2023

HenryRWinterbottom requested review from aerorahul and KateFriedman-NOAA November 17, 2023 20:59

HenryRWinterbottom marked this pull request as ready for review November 17, 2023 21:00

KateFriedman-NOAA requested changes Nov 17, 2023

View reviewed changes

WalterKolczynski-NOAA requested changes Nov 20, 2023

View reviewed changes

HenryRWinterbottom mentioned this pull request Nov 20, 2023

Adds NPOESS support to global-workflow #2083

Merged

7 tasks

WalterKolczynski-NOAA requested changes Nov 21, 2023

View reviewed changes

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved

workflow/rocoto/gfs_tasks.py Outdated Show resolved Hide resolved

WalterKolczynski-NOAA self-requested a review November 29, 2023 21:40

HenryRWinterbottom closed this Nov 29, 2023

HenryRWinterbottom force-pushed the feature/gfsv17_issue_1219 branch from 2e4a020 to c19810c Compare November 29, 2023 21:43

This was referenced Nov 29, 2023

Adds GDAS GEMPAK jobs to rocoto task mesh #2103

Closed

Adds GDAS GEMPAK jobs to rocoto task mesh #2126

Closed

Adds GDAS GEMPAK jobs to rocoto task mesh #2127

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds GDAS GEMPAK jobs to rocoto task mesh #2073

Adds GDAS GEMPAK jobs to rocoto task mesh #2073

HenryRWinterbottom commented Nov 16, 2023 •

edited

Loading

HenryRWinterbottom commented Nov 16, 2023

HenryRWinterbottom commented Nov 16, 2023

WalterKolczynski-NOAA commented Nov 16, 2023

HenryRWinterbottom commented Nov 17, 2023

WalterKolczynski-NOAA commented Nov 17, 2023

KateFriedman-NOAA left a comment

KateFriedman-NOAA Nov 17, 2023

KateFriedman-NOAA Nov 17, 2023

WalterKolczynski-NOAA Nov 20, 2023

HenryRWinterbottom Nov 20, 2023

KateFriedman-NOAA Nov 17, 2023

HenryRWinterbottom Nov 20, 2023

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Nov 20, 2023

HenryRWinterbottom Nov 20, 2023

HenryRWinterbottom Nov 20, 2023

WalterKolczynski-NOAA commented Nov 29, 2023

Adds GDAS GEMPAK jobs to rocoto task mesh #2073

Adds GDAS GEMPAK jobs to rocoto task mesh #2073

Conversation

HenryRWinterbottom commented Nov 16, 2023 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

HenryRWinterbottom commented Nov 16, 2023

HenryRWinterbottom commented Nov 16, 2023

WalterKolczynski-NOAA commented Nov 16, 2023

HenryRWinterbottom commented Nov 17, 2023

WalterKolczynski-NOAA commented Nov 17, 2023

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Nov 29, 2023

HenryRWinterbottom commented Nov 16, 2023 •

edited

Loading