Create Gaussian sfcanl file at the beginning of IAU window #4049

EricSinsky-NOAA · 2025-09-11T18:50:35Z

Description

This PR creates a Gaussian sfcanl file at the beginning of the IAU window. This is accomplished by modifying ush/gaussian_sfcanl.sh in the analcalc task. This file is created only when DOIAU is YES.
This PR will remain a draft until further tests are performed. The archive yamls are also modified so that this new file is added to the GFS archive.

Resolves #3895

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

The C96C48mx500_S2SW_cyc_gfs test has been performed on WCOSS2.

Checklist

ush/gaussian_sfcanl.sh

EricSinsky-NOAA · 2025-09-16T12:25:44Z

Marking this PR as ready for review. All preliminary offline CI GFS tests passed on WCOSS2 except for the gfs_gempakmeta task in C48_S2SW_extended, which is being investigated.

******** C48ATM ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C48_ATM
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 15 2025 13:45:39    Sep 15 2025 15:45:26
******** C48mx5003DVarAOWCDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C48mx500_3DVarAOWCDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 15 2025 13:45:33    Sep 15 2025 14:05:29
202103250000        Done    Sep 15 2025 13:45:33    Sep 15 2025 15:55:35
******** C48mx500hybAOWCDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C48mx500_hybAOWCDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Sep 15 2025 13:45:29    Sep 15 2025 14:05:36
202103250000        Done    Sep 15 2025 13:45:29    Sep 15 2025 15:45:41
******** C48S2SW ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C48_S2SW
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Sep 15 2025 13:45:16    Sep 15 2025 15:45:49
******** C48S2SWextended ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C48_S2SW_extended
202103231200          gfs_gempakmeta                   218271868                DEAD                 271         2         219.0
******** C96atm3DVarextended ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96_atm3DVar_extended
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 15 2025 13:45:31    Sep 15 2025 14:05:29
202112210000        Done    Sep 15 2025 13:45:31    Sep 15 2025 19:25:39
202112210600        Done    Sep 15 2025 13:45:31    Sep 15 2025 21:35:41
202112211200        Done    Sep 15 2025 14:10:34    Sep 15 2025 23:10:32
202112211800        Done    Sep 15 2025 19:30:21    Sep 16 2025 03:40:37
******** C96C48hybatmDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96C48_hybatmDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Sep 15 2025 13:45:34    Sep 15 2025 14:05:38
202112210000        Done    Sep 15 2025 13:45:34    Sep 15 2025 16:35:37
202112210600        Done    Sep 15 2025 13:45:34    Sep 15 2025 16:10:38
******** C96C48hybatmsnowDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96C48_hybatmsnowDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 15 2025 13:45:24    Sep 15 2025 14:05:29
202112201800        Done    Sep 15 2025 13:45:24    Sep 15 2025 16:15:42
202112210000        Done    Sep 15 2025 13:45:24    Sep 15 2025 16:05:39
******** C96C48hybatmsoilDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96C48_hybatmsoilDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202205150600        Done    Sep 15 2025 13:45:50    Sep 15 2025 14:05:30
202205151200        Done    Sep 15 2025 13:45:50    Sep 15 2025 16:35:36
202205151800        Done    Sep 15 2025 13:45:50    Sep 15 2025 16:50:28
******** C96C48mx500S2SWcycgfs ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96C48mx500_S2SW_cyc_gfs
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Sep 15 2025 13:45:46    Sep 15 2025 14:05:45
202112201800        Done    Sep 15 2025 13:45:46    Sep 15 2025 16:30:51
202112210000        Done    Sep 15 2025 13:45:46    Sep 15 2025 18:05:35
202112211800        Done    Sep 15 2025 14:10:37    Sep 15 2025 17:55:51
******** C96C48ufshybatmDA ********
/lfs/h2/emc/ens/noscrub/eric.sinsky/GEFS/expdir/issue3895/EXPDIR/C96C48_ufs_hybatmDA
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Sep 15 2025 13:45:34    Sep 15 2025 14:05:29
202402240000        Done    Sep 15 2025 13:45:34    Sep 15 2025 17:05:34
202402240600        Done    Sep 15 2025 13:45:34    Sep 15 2025 16:40:38

aerorahul · 2025-09-16T13:34:21Z

ush/gaussian_sfcanl.sh

+  iy=${gcycle_date:0:4}
+  im=${gcycle_date:4:2}
+  id=${gcycle_date:6:2}
+  ih=${gcycle_date:8:2}

-# input analysis tiles (with nst records)
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile1.nc" "./anal.tile1.nc"
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile2.nc" "./anal.tile2.nc"
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile3.nc" "./anal.tile3.nc"
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile4.nc" "./anal.tile4.nc"
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile5.nc" "./anal.tile5.nc"
-${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile6.nc" "./anal.tile6.nc"
+  export OMP_NUM_THREADS=${OMP_NUM_THREADS_SFC:-1}


Suggested change

iy=${gcycle_date:0:4}

im=${gcycle_date:4:2}

id=${gcycle_date:6:2}

ih=${gcycle_date:8:2}

# input analysis tiles (with nst records)

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile1.nc" "./anal.tile1.nc"

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile2.nc" "./anal.tile2.nc"

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile3.nc" "./anal.tile3.nc"

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile4.nc" "./anal.tile4.nc"

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile5.nc" "./anal.tile5.nc"

${NLN} "${COMIN_ATMOS_RESTART}/${PDY}.${cyc}0000.sfcanl_data.tile6.nc" "./anal.tile6.nc"

export OMP_NUM_THREADS=${OMP_NUM_THREADS_SFC:-1}

Move this section closer to where it is used, and no need to set OMP_NUM_THREADS each time in the loop.

ush/gaussian_sfcanl.sh

aerorahul · 2025-09-16T13:39:39Z

ush/gaussian_sfcanl.sh

+  gcycle_dates+=("${BDATE}")
+fi
+
+for hr in "${!gcycle_dates[@]}"; do


Suggested change

for hr in "${!gcycle_dates[@]}"; do

for ii in "${!gcycle_dates[@]}"; do

This is an index, not hour.

aerorahul · 2025-09-16T13:39:51Z

ush/gaussian_sfcanl.sh

+
+for hr in "${!gcycle_dates[@]}"; do
+
+  gcycle_date=${gcycle_dates[hr]}


Suggested change

gcycle_date=${gcycle_dates[hr]}

gcycle_date=${gcycle_dates[ii]}

aerorahul · 2025-09-16T13:40:36Z

ush/gaussian_sfcanl.sh

-fi
+  export err=$?
+  if [[ ${err} -ne 0 ]]; then
+    echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status!"


Suggested change

echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status!"

echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status processing ${gcycle_date}!"

aerorahul

Thanks for this PR @EricSinsky-NOAA
A few suggestions if you wouldn't mind taking into consideration.

ush/gaussian_sfcanl.sh

aerorahul · 2025-09-16T13:44:36Z

ush/gaussian_sfcanl.sh

 ################################################################################
 #  Postprocessing
-cd "${pwd}"
+cd "${pwd}" || exit


Suggested change

cd "${pwd}" || exit

cd "${DATA}" || exit

EricSinsky-NOAA · 2025-09-16T14:26:47Z

Thank you for the review, @aerorahul. I have committed and pushed your suggestions.

aerorahul · 2025-09-16T14:28:45Z

ush/gaussian_sfcanl.sh

+  if [[ ${err} -ne 0 ]]; then
+    echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status processing ${gcycle_date}!"
+    # Since we are in "${DATA}/gausfcanl_${gcycle_date:8:2}", we need to step out
+    cd "${DATA}" || exit "${err}"


Suggested change

cd "${DATA}" || exit "${err}"

exit "${err}"

This is incorrect, as cd ${DATA} will be true, and the || will not be executed.

aerorahul · 2025-09-16T20:52:14Z

ush/gaussian_sfcanl.sh

+  if [[ ! -d "${DATA}/gausfcanl_${gcycle_date:8:2}" ]]; then
+    mkdir -p "${DATA}/gausfcanl_${gcycle_date:8:2}" 
+  fi
+
+  cd "${DATA}/gausfcanl_${gcycle_date:8:2}"


Suggested change

if [[ ! -d "${DATA}/gausfcanl_${gcycle_date:8:2}" ]]; then

mkdir -p "${DATA}/gausfcanl_${gcycle_date:8:2}"

fi

cd "${DATA}/gausfcanl_${gcycle_date:8:2}"

rm -rf "${DATA}/gausfcanl_${gcycle_date:8:2}"

mkdir -p "${DATA}/gausfcanl_${gcycle_date:8:2}"

cd "${DATA}/gausfcanl_${gcycle_date:8:2}"

We should perhaps start with a clean directory each time, so that when/if the job fails and restarts, it gets a clean directory.

aerorahul

Looks good.
Just one suggestion on starting with a clean directory for the gcycle_date each time.
Ready to run the tests after it.

EricSinsky-NOAA · 2025-09-17T12:43:16Z

Thank you, @aerorahul. I have also requested a review from @CatherineThomas-NOAA. Would it be possible to hold off on CI testing until she’s had a chance to review?

aerorahul

looks good. thanks for addressing the comments and for this work.

CatherineThomas-NOAA

Thanks for putting this together @EricSinsky-NOAA and running the tests. I did a spot check of the new output from the low res v17 tests and it looks reasonable.

One consideration for both @aerorahul @EricSinsky-NOAA : the new file is called gdas.tHHz.sfcanl.nc with HH being the time at the beginning of the window. For the 3D fields, we have analyses and increment files valid at the beginning of the window with the nomenclature of gdas.tHHz.atmi003.nc and gdas.tHHz.atma003.ensres.nc where the HH is the cycle and 003 signifies they are valid at the beginning of the window. What do you think about naming this new file gdas.tHHz.sfca003.nc?

aerorahul · 2025-09-17T17:00:15Z

Thanks for putting this together @EricSinsky-NOAA and running the tests. I did a spot check of the new output from the low res v17 tests and it looks reasonable.

One consideration for both @aerorahul @EricSinsky-NOAA : the new file is called gdas.tHHz.sfcanl.nc with HH being the time at the beginning of the window. For the 3D fields, we have analyses and increment files valid at the beginning of the window with the nomenclature of gdas.tHHz.atmi003.nc and gdas.tHHz.atma003.ensres.nc where the HH is the cycle and 003 signifies they are valid at the beginning of the window. What do you think about naming this new file gdas.tHHz.sfca003.nc?

@CatherineThomas-NOAA Good catch there.
The HH in gdas.tHHz. should always be the cyc and not the hour of the beginning of the window. I support gdas.tcycz.sfca003.nc for surface analysis valid at the beginning of the window.

EricSinsky-NOAA · 2025-09-17T17:27:23Z

Thanks for the suggestion @CatherineThomas-NOAA! I'll work on changing the file names to gdas.tcycz.sfca003.nc. I believe this naming convention will also reduce confusion.

EricSinsky-NOAA · 2025-09-17T19:53:29Z

Converting this PR back to draft since options are being explored to see if the creation of an sfcanl gaussian file at the beginning of the IAU window can be bypassed. The status of this PR will be updated from draft as soon as a decision is made.

EricSinsky-NOAA · 2025-10-08T10:43:28Z

Closing this PR since a sfcanl file at the beginning of the IAU window is no longer needed.

EricSinsky-NOAA added 4 commits September 10, 2025 19:39

Create Gaussian sfcanl file for multiple dates

ffe6b7a

Modify comment

d8ade8d

Define gcycle_dates

811d2b2

Fix spacing

2555e2b

github-advanced-security bot found potential problems Sep 11, 2025

View reviewed changes

ush/gaussian_sfcanl.sh Fixed Show fixed Hide fixed

EricSinsky-NOAA added 7 commits September 11, 2025 19:19

Address shellnorms

574b191

Remove unneeded cd

96cb60c

Add second sfcanl file to the archive

5a506c4

Address shellnorms

85ab83e

Merge remote-tracking branch 'origin/develop' into issue3895

4376702

Fix indentation

8093cc9

Replace PDY with gcycle_date

d22c6f2

EricSinsky-NOAA marked this pull request as ready for review September 16, 2025 12:25

EricSinsky-NOAA requested a review from aerorahul as a code owner September 16, 2025 12:25

EricSinsky-NOAA requested a review from CatherineThomas-NOAA September 16, 2025 12:26

Merge remote-tracking branch 'origin/develop' into issue3895

904a59b

aerorahul reviewed Sep 16, 2025

View reviewed changes

aerorahul requested changes Sep 16, 2025

View reviewed changes

aerorahul reviewed Sep 16, 2025

View reviewed changes

ush/gaussian_sfcanl.sh Show resolved Hide resolved

aerorahul reviewed Sep 16, 2025

View reviewed changes

EricSinsky-NOAA added 6 commits September 16, 2025 14:01

Move OMP_NUM_THREADS outside loop

20b44db

Remove redundant variable naming

2ee2426

Change hr to ii

00a3376

Give more specific fatal error message

b6a4e61

cd to $DATA before exiting with or without error

cb780fc

Remove unused $pwd variable

1558868

aerorahul reviewed Sep 16, 2025

View reviewed changes

EricSinsky-NOAA added 2 commits September 16, 2025 14:32

Fix error exiting

32cbd3f

Remove comment

168bc98

aerorahul reviewed Sep 16, 2025

View reviewed changes

Remove and remake DATA subdirectory

3b5b17c

aerorahul previously approved these changes Sep 17, 2025

View reviewed changes

CatherineThomas-NOAA requested changes Sep 17, 2025

View reviewed changes

Change the name of sfcanl Gaussian file

d27663d

EricSinsky-NOAA dismissed aerorahul’s stale review via d27663d September 17, 2025 19:39

EricSinsky-NOAA marked this pull request as draft September 17, 2025 19:53

EricSinsky-NOAA closed this Oct 8, 2025

	for hr in "${!gcycle_dates[@]}"; do
	for ii in "${!gcycle_dates[@]}"; do


		for hr in "${!gcycle_dates[@]}"; do

		gcycle_date=${gcycle_dates[hr]}

	gcycle_date=${gcycle_dates[hr]}
	gcycle_date=${gcycle_dates[ii]}

	echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status!"
	echo "FATAL ERROR: ${GAUSFCANLEXE} returned non-zero exit status processing ${gcycle_date}!"

Create Gaussian sfcanl file at the beginning of IAU window #4049

Create Gaussian sfcanl file at the beginning of IAU window #4049

Uh oh!

Conversation

EricSinsky-NOAA commented Sep 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

EricSinsky-NOAA commented Sep 16, 2025

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

EricSinsky-NOAA commented Sep 16, 2025

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul Sep 16, 2025

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

EricSinsky-NOAA commented Sep 17, 2025

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

CatherineThomas-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul commented Sep 17, 2025

Uh oh!

EricSinsky-NOAA commented Sep 17, 2025

Uh oh!

EricSinsky-NOAA commented Sep 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

EricSinsky-NOAA commented Oct 8, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

EricSinsky-NOAA commented Sep 11, 2025 •

edited

Loading

EricSinsky-NOAA commented Sep 17, 2025 •

edited

Loading