Various wave updates: Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid #3190

JessicaMeixner-NOAA · 2024-12-20T22:14:54Z

Description

This PR adds the following:

converting from inp -> nml (@sbanihash)
turning on PIO for waves for restarts (@sbanihash)
enabling cycling for WW3 which required some updates to wave prep jobs + changing what restarts are being saved/etc
changed the way CEMPS, MOM6, CICE and WW3 write restarts to be in sync with FV3 for IAU, which required moving the ufs-weather-model forward one hash to use the new flexible restart feature. (UFS PR Flexible restart write times (restart_fh) for med, ocn, ice, wav ufs-community/ufs-weather-model#2419)
adds uglo_15km - the targeted new wave grid.

Note this PR requires the following:

update to fix files to add uglo_15km
staging ICs for high resolution test case for uglo_15km

Co-author: @sbanihash

Related Issues:

Fixes updates from RTs for wave restart name and nml files #1457
Fixes Update to use new WW3 grid, targeted for GFSv17 #3154
Fixes Cycled waves for GFSv17 #1795
related to Enable IAU across components in the coupled model #1776

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? YES/NO
Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model Flexible restart write times (restart_fh) for med, ocn, ice, wav ufs-community/ufs-weather-model#2419 (Already merged)
- wxflow

How has this been tested?

Tests were done by adding waves to C48mx500 3DVar tests for cycling.
High resolution tests cases were also added.

This likely needs a new round of testing before CI and after staged ICs & new fix files are added.

Checklist

Conflicts: .gitmodules

…reading if desired

…IOswitch

…combineforww3testing

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

aerorahul · 2024-12-31T15:41:26Z

ush/parsing_model_configure_FV3.sh

@@ -25,6 +25,7 @@ local SHOUR=${model_start_date:8:2}
 local FHROT=${IAU_FHROT:-0}
 local DT_ATMOS=${DELTIM}
 local RESTART_INTERVAL="${FV3_RESTART_FH[*]}"
+local RESTART_FH="${CMEPS_RESTART_FH:-" "}"


I am not sure introducing a CMEPS variable dependency in FV3 would be clean, since that means there is a dependency on a coupled model variable for an atm-only configuration. Need to think a bit on this.

The RESART_FH variable is not used by FV3, however it is in the model_configure file.

Also - please note this is what is being done in this PR: ufs-community/ufs-weather-model#2419 so if you want changes we'll potentially need changes to ufs-weather-model

If the FV3 does not use this variable, can we just set this as:
local RESTART_FH=" "

This assumes the restarts are written according to local RESTART_INTERVAL="${FV3_RESTART_FH[*]}", correct?

I misunderstood this comment that said the FV3 does not use this variable. Does the model_configure also now use this variable to specify restart write times for all model components? (In the past, model_configure was fv3atm specific)
How does this relate to RESTART_INTERVAL that is currently specifying the FV3 restart times? Does RESTART_FH override RESTART_INTERVAL?

From @NickSzapiro-NOAA in an email:

3 restart triggers: restart_n+restart_option: frequency-based ESMF restart trigger (via ufs.configure) *Not currently in FV3 restart_interval: just in FV3 (via model_configure) and can mean different things based on "encoding" restart_fh: UFS wraps ESMF trigger (via model_configure). *Not currently in FV3

@aerorahul - let me know if you'd like me to run a test case and save the output showing that we're writing all restarts at the same time when using IAU for all components. I can also run an fv3 standalone ci test to ensure that we are not negatively impacting atm-only runs too.

@aerorahul - is there any change you'd like to see with this? Or any specific tests you'd like to see output with?

No. I am ok with the movement of CMEPS_RESTART_FH into the CMEPS_predet function.

aerorahul · 2024-12-31T15:43:03Z

ush/forecast_predet.sh

@@ -134,6 +134,29 @@ common_predet(){
  # Several model components share DATA/INPUT for input data
  if [[ ! -d "${DATA}/INPUT" ]]; then mkdir -p "${DATA}/INPUT"; fi

+  # For CMEPS, CICE, MOM6 and WW3 determine restart writes
+  # Note FV3 has its own restart intervals  
+  cmeps_restart_interval=${restart_interval:-${FHMAX}}


This isn't a common variable if it starts w/ cmeps.

What would you prefer it be called? This is for CMEPS, WW3, ICE, and MOM6

It is not the name, but the place these are introduced.
The function CMEPS_predet IMO, would be appropriate to introduce these variables related to cmeps_ (and more general the coupled model since they share these concepts)

Thanks for the additional explanation. I will move it to CMEPS!

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

JessicaMeixner-NOAA · 2025-01-03T19:24:50Z

ush/forecast_predet.sh

+      local restart_interval_start=${cmeps_restart_interval}
+      local restart_interval_end=${FHMAX}
+    fi
+    CMEPS_RESTART_FH="$(seq -s ' ' "${restart_interval_start}" "${cmeps_restart_interval}" "${restart_interval_end}")"


Any recommendations on how to fix this @DavidHuber-NOAA or @aerorahul ?

Add

# shellcheck disable=SC2034

before the function definition before line 731. See MOM6_predet for example.

aerorahul

I have no concerns on this PR. Just a few comments and answers to questions.
Thanks for enabling the fortran namelist as well as using the template from ufs-weather-model.

aerorahul · 2025-01-03T23:51:28Z

ush/forecast_postdet.sh

    fi
  fi

  # Link restart files
  for (( vdate = first_ww3_restart_out; vdate <= forecast_end_cycle;
         vdate = $(date --utc -d "${vdate:0:8} ${vdate:8:2} + ${restart_interval} hours" +%Y%m%d%H) )); do
-    ww3_restart_file="${vdate:0:8}.${vdate:8:2}0000.restart.ww3"
-    ${NLN} "${DATArestart}/WW3_RESTART/${ww3_restart_file}" "${ww3_restart_file}"
+	 seconds=$(to_seconds "${vdate:8:2}0000")  # convert HHMMSS to seconds


Suggested change

seconds=$(to_seconds "${vdate:8:2}0000") # convert HHMMSS to seconds

seconds=$(to_seconds "${vdate:8:2}0000") # convert HHMMSS to seconds

aerorahul · 2025-01-03T23:54:01Z

ush/forecast_postdet.sh

-  # TODO: Need to add logic to copy restarts from DATArestart/WW3_RESTART to COMOUT_WAVE_RESTART
+
+  # Copy wave namelist from DATA to COMOUT_CONF after the forecast is run (and successfull)
+  ${NCP} "${DATA}/ww3_shel.nml" "${COMOUT_CONF}/ufs.ww3_shel.nml"


This is fine, but beware that in seg1 will overwrite seg0's namelist.

aerorahul · 2025-01-03T23:56:39Z

ush/forecast_predet.sh

+      local restart_interval_start=${cmeps_restart_interval}
+      local restart_interval_end=${FHMAX}
+    fi
+    CMEPS_RESTART_FH="$(seq -s ' ' "${restart_interval_start}" "${cmeps_restart_interval}" "${restart_interval_end}")"


Add

# shellcheck disable=SC2034

before the function definition before line 731. See MOM6_predet for example.

aerorahul · 2025-01-04T00:00:07Z

ush/parsing_model_configure_FV3.sh

@@ -25,6 +25,7 @@ local SHOUR=${model_start_date:8:2}
 local FHROT=${IAU_FHROT:-0}
 local DT_ATMOS=${DELTIM}
 local RESTART_INTERVAL="${FV3_RESTART_FH[*]}"
+local RESTART_FH="${CMEPS_RESTART_FH:-" "}"


No. I am ok with the movement of CMEPS_RESTART_FH into the CMEPS_predet function.

aerorahul · 2025-01-04T00:01:08Z

ush/parsing_namelists_WW3.sh

+  local WW3_OUTPARS=$OUTPARS_WAV 
+  local WW3_DTFLD=$DTFLD_WAV
+  local WW3_DTPNT=$DTPNT_WAV


Suggested change

local WW3_OUTPARS=$OUTPARS_WAV

local WW3_DTFLD=$DTFLD_WAV

local WW3_DTPNT=$DTPNT_WAV

local WW3_OUTPARS="${OUTPARS_WAV}"

local WW3_DTFLD="${DTFLD_WAV}"

local WW3_DTPNT="${DTPNT_WAV}"

Not sure why shellnorms did not pick up on this.

aerorahul · 2025-01-04T00:02:17Z

ush/parsing_namelists_WW3.sh

 # Buoy location file

  if [ -f ${PARMgfs}/wave/wave_${NET}.buoys ]
  then
-    cp ${PARMgfs}/wave/wave_${NET}.buoys buoy.loc
+    cp ${PARMgfs}/wave/wave_${NET}.buoys ${DATA}/ww3_points.list


Suggested change

cp ${PARMgfs}/wave/wave_${NET}.buoys ${DATA}/ww3_points.list

${NCP} "${PARMgfs}/wave/wave_${NET}.buoys" "${DATA}/ww3_points.list"

aerorahul · 2025-01-04T00:06:51Z

workflow/rocoto/gfs_tasks.py

@@ -950,10 +950,6 @@ def _fcst_cycled(self):
            dependencies.append(rocoto.add_dependency(dep_dict))
            dependencies = rocoto.create_dependency(dep_condition='or', dep=dependencies)



There are 2 blank lines here. This is likely causing the pynorms test to fail

JessicaMeixner-NOAA and others added 30 commits July 15, 2024 16:55

updates for uglo 15km

54dd21b

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

5069875

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

0791f27

Merge remote-tracking branch 'EMC/develop' into feature/uglo_15km

dfbdf63

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

2685096

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

37ae897

Merge remote-tracking branch 'EMC/develop' into feature/uglo_15km

fcc0979

update wave grids in gfs/config.base

ffea11d

update build and UFS model

2b33cd5

update ush_configure for wave related variables

a3d9b4c

Merge branch 'NOAA-EMC:develop' into bug/addPIOswitch

1149faf

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

67b5f54

Merge remote-tracking branch 'EMC/develop' into bug/addPIOswitch

c470488

Conflicts: .gitmodules

update for CICE

b575ff6

update ufs to version with fix for waves

8020fc7

remove output every time-step

94f9f2a

Merge branch 'NOAA-EMC:develop' into bug/addPIOswitch

e3a1bd9

update link script for UPP update

886e638

compute resources when doing traditional threading

cf21059

fix shel

32ec7b2

tasks_per_node should know about threads

d5335a1

spacing

bdce86f

add QUILTING_RESTART as an option

0ee666d

update config.resources in GEFS system to allow use of traditional th…

26a0c42

…reading if desired

ww3_shel.inp >> ww3_shel.nml

89225c5

Merge branch 'NOAA-EMC:develop' into feature/uglo_15km

6eb5778

Merge remote-tracking branch 'origin/feature/uglo_15km' into bug/addP…

6d5eb04

…IOswitch

Merge remote-tracking branch 'RM/feature/traditional_threading' into …

d526db6

…combineforww3testing

testing temp files

47be8b3

update ufs

0be2ee2

DavidHuber-NOAA changed the title ~~Various wave updates~~ Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid Dec 23, 2024

JessicaMeixner-NOAA and others added 18 commits December 30, 2024 16:14

Merge branch 'NOAA-EMC:develop' into enablepio_cycle_uglo15km

4097a37

Update parm/config/gefs/config.ufs

6741814

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update parm/config/gfs/config.ufs

d0d4661

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update parm/config/gfs/config.ufs

4ff02a2

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update parm/config/gfs/config.wave

b74305a

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_postdet.sh

77e24e2

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_postdet.sh

b09ddff

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_postdet.sh

b20d9a7

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_postdet.sh

0ab6a40

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

a4721c0

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

6ff0ce7

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

cb97cf6

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

300c1de

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

b1b7cd0

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_predet.sh

521f4d3

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/forecast_postdet.sh

cdc5a9c

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update ush/parsing_ufs_configure.sh

a381d54

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update workflow/rocoto/gfs_tasks.py

951a117

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Update wave.rst

e889719

JessicaMeixner-NOAA mentioned this pull request Dec 31, 2024

Clone, build, and run C48_ATM and C48_S2SW on Gaea C5 and C6 #3106

Open

19 tasks

aerorahul reviewed Dec 31, 2024

View reviewed changes

JessicaMeixner-NOAA and others added 3 commits January 3, 2025 08:59

Update ush/forecast_postdet.sh

49ff12d

Co-authored-by: David Huber <69919478+DavidHuber-NOAA@users.noreply.github.com>

Merge remote-tracking branch 'EMC/develop' into enablepio_cycle_uglo15km

84989e6

move code from common to cmeps for restart variables

db8f725

github-advanced-security bot found potential problems Jan 3, 2025

View reviewed changes

aerorahul reviewed Jan 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Various wave updates: Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid #3190

Various wave updates: Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid #3190

JessicaMeixner-NOAA commented Dec 20, 2024 •

edited

Loading

aerorahul Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

aerorahul Dec 31, 2024

aerorahul Dec 31, 2024

aerorahul Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

JessicaMeixner-NOAA Jan 3, 2025

aerorahul Jan 4, 2025

aerorahul Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

aerorahul Dec 31, 2024

JessicaMeixner-NOAA Dec 31, 2024

JessicaMeixner-NOAA Jan 3, 2025

JessicaMeixner-NOAA Jan 3, 2025

aerorahul Jan 3, 2025

aerorahul left a comment

aerorahul Jan 3, 2025

aerorahul Jan 3, 2025

aerorahul Jan 3, 2025

aerorahul Jan 4, 2025

aerorahul Jan 4, 2025

aerorahul Jan 4, 2025

aerorahul Jan 4, 2025

	seconds=$(to_seconds "${vdate:8:2}0000") # convert HHMMSS to seconds
	seconds=$(to_seconds "${vdate:8:2}0000") # convert HHMMSS to seconds

	cp ${PARMgfs}/wave/wave_${NET}.buoys ${DATA}/ww3_points.list
	${NCP} "${PARMgfs}/wave/wave_${NET}.buoys" "${DATA}/ww3_points.list"

		@@ -950,10 +950,6 @@ def _fcst_cycled(self):
		dependencies.append(rocoto.add_dependency(dep_dict))
		dependencies = rocoto.create_dependency(dep_condition='or', dep=dependencies)

Various wave updates: Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid #3190

Are you sure you want to change the base?

Various wave updates: Reformat WW3 inputs; enable PIO for wave restarts; enable WW3 cycling; synchronize UFS restart writes; add uglo_15km WW3 grid #3190

Conversation

JessicaMeixner-NOAA commented Dec 20, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JessicaMeixner-NOAA commented Dec 20, 2024 •

edited

Loading