Skip to content

Save marine LETKF yaml #3489

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jun 6, 2025
Merged

Save marine LETKF yaml #3489

merged 5 commits into from
Jun 6, 2025

Conversation

shlyaeva
Copy link
Contributor

Description

Saves marine LETKF yaml for posterity and for EVA verifications plots.

Note: this is not urgent as it's not required for v17.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? NO

How has this been tested?

Tested with the hybrid C384m025 experiment in ci/cases/gfsv17, with hera/intel. The yaml is saved in COMROOT as expected and successfully used for diagnostics plots (see NOAA-EMC/gdas-marine-viz#511 for an example).

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have documented my code, including function, input, and output descriptions
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • This change is covered by an existing CI test or a new one has been added
  • Any new scripts have been added to the .github/CODEOWNERS file with owners
  • I have made corresponding changes to the system documentation if necessary

AndrewEichmann-NOAA
AndrewEichmann-NOAA previously approved these changes Mar 25, 2025
View reviewed changes
Copy link
Contributor

@AndrewEichmann-NOAA AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

aerorahul
aerorahul reviewed Mar 25, 2025
View reviewed changes
ush/python/pygfs/task/marine_letkf.py Outdated
@@ -205,5 +205,9 @@ def finalize(self):
# stage the desired diag files
FileHandler({'mkdir': [os.path.join(letkfsaveconf.COMOUT_OCEAN_LETKF, 'diags')]}).sync()
FileHandler({'copy': obs_files_to_copy}).sync()
# yaml configurations
yamls_to_copy = []
yamls_to_copy.append([letkfsaveconf.letkf_yaml_file, os.path.join(letkfsaveconf.COMOUT_OCEAN_LETKF, 'letkf.yaml')])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This action is fine.
However, we are saving all the configurations to directory named COM_CONF instead of COMOUT_OCEAN_LETKF. COM_CONF is a directory in COM that has all the configurations of all the executables that are run in a cycle.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, I didn't know this! I'll take a look later this week to change this and maybe also some other marine yamls that get copied to a different directory.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Finally getting back to this. Is there a place where things from COM_CONF get archived? I haven't found it in the code yet. I'm wondering if I need a custom addition for archiving this yaml, or if it's already handled some place.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

E.g.

global-workflow/parm/archive/gfsa.yaml.j2

Lines 14 to 15 in b1c60d4

# UFS configuration
- "{{ COMIN_CONF | relpath(ROTDIR) }}/ufs.input.nml"

shows that the gfsa.tar will contain the ufs.input.nml from COM_CONF.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A potential place to add this file to the archive list would be in this yaml.

guillaumevernieres
guillaumevernieres previously approved these changes Apr 3, 2025
View reviewed changes
Copy link
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@shlyaeva shlyaeva marked this pull request as draft April 3, 2025 20:18
@DavidNew-NOAA DavidNew-NOAA added the JEDI Feature development to support JEDI-based DA label Apr 15, 2025
@shlyaeva shlyaeva marked this pull request as ready for review June 3, 2025 21:44
@shlyaeva shlyaeva requested a review from aerorahul June 3, 2025 21:45
aerorahul
aerorahul requested changes Jun 4, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rest of the changes looks ok, but I have one comment that needs your attention.

@shlyaeva shlyaeva requested a review from aerorahul June 5, 2025 17:49
aerorahul
aerorahul approved these changes Jun 5, 2025
View reviewed changes
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks @shlyaeva

@aerorahul aerorahul added the CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 label Jun 5, 2025
@emcbot emcbot added CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 CI-Gaeac6-Running labels Jun 5, 2025
DavidHuber-NOAA
DavidHuber-NOAA reviewed Jun 6, 2025
View reviewed changes
parm/archive/master_enkf.yaml.j2
Comment on lines +77 to +82
{% set tmpl_dict = ({ '${ROTDIR}':ROTDIR,
'${RUN}':RUN,
'${YMD}':cycle_YMD,
'${HH}':cycle_HH,
'${MEMDIR}': "ensstat"}) %}
{% set COMIN_CONF = COM_CONF_TMPL | replace_tmpl(tmpl_dict) %}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should be moved to JGLOBAL_ENS_ARCHIVE_TARS by inserting a declaration here:

global-workflow/jobs/JGLOBAL_ENS_ARCHIVE_TARS

Lines 9 to 15 in e131fb1

MEMDIR="ensstat" YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \
COMIN_ATMOS_ANALYSIS_ENSSTAT:COM_ATMOS_ANALYSIS_TMPL \
COMIN_ATMOS_HISTORY_ENSSTAT:COM_ATMOS_HISTORY_TMPL \
COMIN_ICE_ANALYSIS_ENSSTAT:COM_ICE_ANALYSIS_TMPL \
COMIN_OCEAN_ANALYSIS_ENSSTAT:COM_OCEAN_ANALYSIS_TMPL \
COMIN_SNOW_ANALYSIS_ENSSTAT:COM_SNOW_ANALYSIS_TMPL \
COMOUT_CONF:COM_CONF_TMPL

But I'm OK with this being handled as part of #3767.

DavidHuber-NOAA
DavidHuber-NOAA approved these changes Jun 6, 2025
View reviewed changes
Copy link
Contributor

@DavidHuber-NOAA DavidHuber-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve with a note on a future change request for the COMIN_CONF declaration in master_enkf.yaml.j2 to be moved to JGLOBAL_ENS_ARCHIVE_TARS.

@DavidHuber-NOAA DavidHuber-NOAA merged commit 311e7f1 into NOAA-EMC:develop Jun 6, 2025
5 checks passed
@shlyaeva shlyaeva mentioned this pull request Jun 6, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aerorahul aerorahul aerorahul approved these changes

@DavidHuber-NOAA DavidHuber-NOAA DavidHuber-NOAA approved these changes

@guillaumevernieres guillaumevernieres guillaumevernieres left review comments

@AndrewEichmann-NOAA AndrewEichmann-NOAA AndrewEichmann-NOAA left review comments

Assignees
No one assigned
Labels
CI-Gaeac6-Passed **Bot use only** CI testing on Gaea C6 for this PR has completed successfully JEDI Feature development to support JEDI-based DA
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@shlyaeva @aerorahul @guillaumevernieres @AndrewEichmann-NOAA @DavidHuber-NOAA @emcbot @DavidNew-NOAA