Add fv3-jedi and its dependencies (SkyLab 8 version) by rhoneyager-tomorrow · Pull Request #395 · JCSDA/spack

rhoneyager-tomorrow · 2024-01-24T20:28:54Z

Description

The previous PR (#306) has been around for a while. I have updated it for the new SkyLab v7 release.

This provides the same as before, with the addition of mom6 and soca packages.

Errata

I've excluded updates/additions of mpas-model and mpas-jedi from this PR. The mpas package on Spack is entirely incompatible with the JCSDA fork. CMake packaging logic vs a custom Makefile. I tested production of a CMake-ified version mpas-model package, but there is a bug in the CMake package export logic in the JCSDA fork, and this prevents use with mpas-jedi.

Checklist

I have performed a self-review of my own code
I have made corresponding changes to the documentation
I have run the unit tests before creating the PR

…kylab7_components

climbfuji · 2024-01-25T15:43:24Z

@rhoneyager-tomorrow This replaces #306, correct?

rhoneyager-tomorrow · 2024-01-25T15:44:09Z

Yes. I can close the old version.

climbfuji · 2024-03-11T16:21:07Z

@rhoneyager-tomorrow Let's see if we can get this PR merged/resolved. My preference would be to apply all the patches that you submitted here to the Skylab v8 release and then do away with anything prior to that in the spack PR. I would also prefer to pick apart this PR and add/update packages individually. For packages that are also in spack mainline, I would prefer submitting the updates to the upstream repo first and then bring them down to our fork.

Other notes:

Hopefully we don't need to maintain our GFDL_atmos_cubed_sphere fork for much longer and can just use the FV3 dycore code from GFDL directly (without the ecbuild stuff on top).
The spack-stack package "spack repo" that used to be var/spack/repos/jcsda_emc/...is now part of the spack-stack GitHub repository in subdirectoryspack-ext`

I can assist with or take care of some or most of this work, but hopefully we can split the work somewhat. Thoughts?

climbfuji · 2024-03-11T16:32:12Z

@rhoneyager-tomorrow I am going to submit this to femps (note that I removed the HINTS since the spack-package adds the external dependency on ecbuild:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index 6b2e041..9f570cb 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -4,6 +4,8 @@

 cmake_minimum_required( VERSION 3.3.2 FATAL_ERROR )

+find_package( ecbuild 3.6 REQUIRED )
+
 project( femps VERSION 1.2.0 LANGUAGES Fortran )

 set( CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake;${CMAKE_MODULE_PATH})
diff --git a/src/femps/CMakeLists.txt b/src/femps/CMakeLists.txt
index f7c2e65..3852a11 100644
--- a/src/femps/CMakeLists.txt
+++ b/src/femps/CMakeLists.txt
@@ -25,3 +25,11 @@ ecbuild_add_library( TARGET          femps

 target_include_directories( femps INTERFACE $<BUILD_INTERFACE:${CMAKE_Fortran_MODULE_DIRECTORY}>
                                             $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
+
+# Fortran module output directory for build and install interfaces
+set( MODULE_DIR module/${PROJECT_NAME}/${CMAKE_Fortran_COMPILER_ID}/${CMAKE_Fortran_COMPILER_VERSION} )
+set_target_properties( ${PROJECT_NAME} PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_BINARY_DIR}/${MODULE_DIR} )
+install( DIRECTORY ${CMAKE_BINARY_DIR}/${MODULE_DIR}/ DESTINATION ${MODULE_DIR} )
+target_include_directories( femps INTERFACE
+                            $<BUILD_INTERFACE:${CMAKE_BINARY_DIR}/${MODULE_DIR}>
+                            $<INSTALL_INTERFACE:${MODULE_DIR}> )

Does that look ok for you?

climbfuji · 2024-03-11T16:48:57Z

Same for fv3-jedi-linearmodel:

diff --git a/CMakeLists.txt b/CMakeLists.txt
index ffa8887..1d4f7f1 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -15,6 +15,8 @@ set( CMAKE_MODULE_PATH ${CMAKE_CURRENT_SOURCE_DIR}/cmake;${CMAKE_MODULE_PATH} )

 set( CMAKE_DIRECTORY_LABELS "fv3-jedi-lm" )

+find_package( ecbuild 3.6 REQUIRED )
+
 set( ECBUILD_DEFAULT_BUILD_TYPE Release )
 set( ENABLE_OS_TESTS           OFF CACHE BOOL "Disable OS tests" FORCE )
 set( ENABLE_LARGE_FILE_SUPPORT OFF CACHE BOOL "Disable testing of large file support" FORCE )
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index 2d320ef..f4e4406 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -40,3 +40,11 @@ endif()
 if(CMAKE_Fortran_COMPILER_ID MATCHES GNU AND CMAKE_Fortran_COMPILER_VERSION VERSION_GREATER_EQUAL 10)
     target_compile_options(fv3jedilm PRIVATE $<$<COMPILE_LANGUAGE:Fortran>:-fallow-argument-mismatch>)
 endif()
+
+# Fortran module output directory for build and install interfaces
+set( MODULE_DIR module/${PROJECT_NAME}/${CMAKE_Fortran_COMPILER_ID}/${CMAKE_Fortran_COMPILER_VERSION} )
+set_target_properties( ${PROJECT_NAME} PROPERTIES Fortran_MODULE_DIRECTORY ${CMAKE_BINARY_DIR}/${MODULE_DIR} )
+install( DIRECTORY ${CMAKE_BINARY_DIR}/${MODULE_DIR}/ DESTINATION ${MODULE_DIR} )
+target_include_directories( fv3jedilm INTERFACE
+                            $<BUILD_INTERFACE:${CMAKE_BINARY_DIR}/${MODULE_DIR}>
+                            $<INSTALL_INTERFACE:${MODULE_DIR}> )

climbfuji · 2024-03-11T16:58:57Z

var/spack/repos/jcsda-emc/packages/fv3-jedi-linearmodel/package.py

+    variant("mpi", default=True, description="Support for MPI distributed parallelism")
+    variant("openmp", default=True, description="Build with OpenMP support")
+
+    conflicts("forecast_model=GEOS", msg="FV3-JEDI-LINEARMODEL: GEOS to be implemented.")


I think we are using the current fv3-jedi-linearmodel with GEOS and UFS - @shlyaeva can you comment on this?

You're using both, but at time of writing this PR GEOS wasn't fully buildable in Spack. Likewise, the ufs-bundle depended on JCSDA-internal, so I didn't have much insight there.

I think we can merge it as-is now, and update later as needed

climbfuji · 2024-03-11T17:00:10Z

var/spack/repos/jcsda-emc/packages/gsw/package.py

+    depends_on("ecbuild", type=("build"))
+    depends_on("ecbuild@3.3.2:", type=("build"), when="@3.0.7:")
+
+    version("3.0.7", commit="1a02ebaf6f7a4e9f2c2d2dd973fb050e697bcc74")


Note the non-standard notation of the tag. In the repo, it's v3.07

climbfuji · 2024-03-11T17:01:38Z

var/spack/repos/jcsda-emc/packages/saber/CMakeLists.txt.patch

+++ b/CMakeLists.txt
+@@ -42,7 +42,7 @@ find_package( atlas 0.33.0 REQUIRED COMPONENTS OMP_Fortran )
+ if( ENABLE_MKL )
+     find_package( MKL )


In fact, this should rather be

if( ENABLE_MKL ) find_package( MKL REQUIRED )

(at least that is what we decided at JCSDA - if you request a feature then it is an error if it is not available)

Actually, we already have the change that you made in develop, ~~and your version makes more sense than what I was thinking initially. That's because it's easier going to use Intel MKL or LAPACK, but if both are missing it will abort.~~ I am going to revert that comment. I will make it a hard requirement if someone asks for MKL.

My idea doesn't work, fixing the underlying problem requires changes across several repositories.

rhoneyager-tomorrow · 2024-03-11T17:18:22Z

@rhoneyager-tomorrow Let's see if we can get this PR merged/resolved. My preference would be to apply all the patches that you submitted here to the Skylab v8 release and then do away with anything prior to that in the spack PR. I would also prefer to pick apart this PR and add/update packages individually. For packages that are also in spack mainline, I would prefer submitting the updates to the upstream repo first and then bring them down to our fork.

SkyLab 8 isn't released yet, but I expect that it will take some time for all the patches to be applied, so I'm quite fine with everything that you propose.

Other notes:

Hopefully we don't need to maintain our GFDL_atmos_cubed_sphere fork for much longer and can just use the FV3 dycore code from GFDL directly (without the ecbuild stuff on top).

Great news! Hopefully you can do the same with mpas eventually.

The spack-stack package "spack repo" that used to be var/spack/repos/jcsda_emc/...is now part of the spack-stack GitHub repository in subdirectoryspack-ext`

I can assist with or take care of some or most of this work, but hopefully we can split the work somewhat. Thoughts?

Splitting works for me. Let me know once you finish reviewing.

climbfuji · 2024-03-11T19:38:24Z

I created the PRs for femps and fv3-jedi-linearmodel so far based on what I posted above

climbfuji · 2024-03-11T22:26:08Z

Regarding the ecmwf-atlas update: version 0.36.0 is already in spack develop, and I just opened spack#43133 to add the missing ecbuild@3.4: dependency. Once that's merged, I can bring both updates down to our fork.

climbfuji · 2024-03-12T16:56:55Z

@rhoneyager-tomorrow We merged the femps and fv3-jedi-linearmodel build system bug fixes. Do you want to check the public develop branches? That will be in the Skylab v8 releases.

climbfuji · 2024-03-12T18:58:31Z

var/spack/repos/builtin/packages/ecmwf-atlas/package.py


    version("master", branch="master")
    version("develop", branch="develop")
+    version("0.36.0", sha256="39bf748aa7b22df80b9791fbb6b4351ed9a9f85587b58fc3225314278a2a68f8")


Changes to this file will no longer be needed. We merged the missing ecbuild@3.4: dependency upstream (spack#43133); spack develop already had 0.36.0. We'll bring both updates to our spack fork shortly.

climbfuji · 2024-03-12T19:03:20Z

var/spack/repos/jcsda-emc/packages/femps/package.py

+    depends_on("netcdf-c")
+    depends_on("netcdf-fortran")
+
+    # find_package(ecbuild REQUIRED) is needed when using ecbuild.


Patches should no longer be required if we target sklyab v8

climbfuji · 2024-03-12T19:04:18Z

var/spack/repos/jcsda-emc/packages/fv3-jedi-linearmodel/package.py

+        res = [self.define_from_variant("FV3_FORECAST_MODEL", "forecast_model")]
+        return res
+
+    # find_package(ecbuild REQUIRED) is needed when using ecbuild.


Patches should no longer be required if we target sklyab v8

climbfuji · 2024-03-13T02:03:28Z

var/spack/repos/jcsda-emc/packages/GFDL_atmos_cubed_sphere/CMakeLists.txt.patch

@@ -0,0 +1,11 @@
+--- a/CMakeLists.txt


No longer required for Skylab v8 because of JCSDA/GFDL_atmos_cubed_sphere#3

climbfuji · 2024-03-14T14:34:07Z

@rhoneyager-tomorrow Here is an update on the PR: I've merged the following patches (w/ slight modifications into the develop code, to be released in Skylab v8 next month):

var/spack/repos/jcsda-emc/packages/GFDL_atmos_cubed_sphere/CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/femps/CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/femps/src.femps.CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi-linearmodel/CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi-linearmodel/src.CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi/CMakeLists.txt.16.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi/CMakeLists.txt.18.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi/cmake.fv3jedi_extra_macros.cmake.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi/test.CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/saber/CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/saber/quench.src.Fields.cc.patch
var/spack/repos/jcsda-emc/packages/saber/saber-import.cmake.in.patch

I also updated the following package recipes in our spack fork:

var/spack/repos/builtin/packages/ecmwf-atlas/package.py (and synced it with spack develop)

What needs to remain in this PR (but updated to only cover the Skylab v8 versions):

var/spack/repos/builtin/packages/crtm/package.py
var/spack/repos/jcsda-emc/packages/GFDL_atmos_cubed_sphere/package.py
var/spack/repos/jcsda-emc/packages/femps/package.py
var/spack/repos/jcsda-emc/packages/fv3-jedi-linearmodel/CMakeLists.txt.patch
var/spack/repos/jcsda-emc/packages/fv3-jedi/package.py
var/spack/repos/jcsda-emc/packages/gsw/package.py (but note that different tag name)
var/spack/repos/jcsda-emc/packages/ioda/package.py
var/spack/repos/jcsda-emc/packages/mom6/package.py
var/spack/repos/jcsda-emc/packages/oops/package.py
var/spack/repos/jcsda-emc/packages/saber/package.py
var/spack/repos/jcsda-emc/packages/soca/package.py
var/spack/repos/jcsda-emc/packages/ufo/package.py
var/spack/repos/jcsda-emc/packages/vader/package.py

climbfuji · 2026-02-13T16:03:26Z

@rickgrubin-noaa @rhoneyager-tomorrow I believe you have whatever JEDI packages you need in Spack at tomorrow.io, correct? With that and my PR JCSDA/spack-stack#1887, should we close this PR for good once JCSDA/spack-stack#1887 is merged?

rickgrubin-noaa · 2026-02-13T22:35:33Z

@rickgrubin-noaa @rhoneyager-tomorrow I believe you have whatever JEDI packages you need in Spack at tomorrow.io, correct? With that and my PR JCSDA/spack-stack#1887, should we close this PR for good once JCSDA/spack-stack#1887 is merged?

Agree with closing this PR once JCSDA/spack-stack#1887 is merged.

This PR updates the JEDI packages from PR JCSDA/spack#395 that are relevant for NEPTUNE-JEDI (aka FALCON project) and adds them to our custom repo under repos/spack_stack. Building these components in spack-stack and providing them as modules allows us to reduce the NEPTUNE-JEDI build to a small number of repositories that are controlled and developed by NRL only, and - very important - allows us to implement a pure CMake build at the top level (neptune-bundle/neptune-ecosystem) without having to deal with ecbuild. A small change is made to the crtm package (in repos/builtin) to enable basic testing during spack install. Notes/Limitations The hashes added here are the exact versions needed by NEPTUNE-JEDI (aka FALCON), with additional bug fixes added on top of it in form of patches applied by Spack. In FALCON, these were kept in branches created from the commits used here. The Spack solution is cleaner, since it avoids maintaining branches. Most of these patches are already included in newer versions of the JEDI code. I made sure that all ctests that are enabled pass with GCC (tested gcc@13) and Intel oneAPI (tested oneapi@2025.3.0). I made sure that the tests that are disabled are not related to the Spack package, but to the code itself or other issues outside of the scope of this PR. To run these tests, use spack install --test=root ioda, for example. FALCON needs exactly these hashes in spack-stack-2.1.0 to transition from building JEDI components in neptune-bundle to loading them as modules. It is understood between JCSDA and NRL that these new Spack packages will not receive any additional support (i.e. beyond what JCSDA supports already) and that we will make an effort to update the JEDI code in the bundle to newer, official tags of the JEDI repositories. We cannot test the JEDI packages in CI yet, because the ropp-ufo code is coming from the NRL Enterprise GitHub because of the unresolved issue Stub with ROPP as submodule JCSDA-internal/ropp-ufo#1.

climbfuji · 2026-02-21T01:41:07Z

Replaced by JCSDA/spack-stack#1887 and code internal to tomorrow.io

rhoneyager-tomorrow and others added 10 commits August 21, 2023 10:51

Add fv3-jedi for Skylab v5.

1939921

Merge branch 'jcsda_emc_spack_stack' into feature/fv3_jedi

6923f36

Updates for spack-stack 1.5 and the SkyLab 6 release.

401763e

black formatting

fa07cbe

Merge branch 'jcsda_emc_spack_stack' into feature/fv3_jedi

30dfeef

flake8 checks

561f9e1

Merge remote-tracking branch 'origin/feature/fv3_jedi' into feature/s…

bbb04d9

…kylab7_components

Updating through fv3-jedi to SkyLab v7

93a01d9

Adding soca to spack build.

c812238

black formatting

a965a72

climbfuji self-requested a review January 25, 2024 15:43

climbfuji self-assigned this Jan 25, 2024

climbfuji added the INFRA JEDI Infrastructure label Jan 25, 2024

climbfuji reviewed Mar 11, 2024

View reviewed changes

climbfuji reviewed Mar 12, 2024

View reviewed changes

This was referenced Mar 12, 2024

ecmwf-atlas@0.36: depend on ecbuild@3.4: (spack PR #43133) #413

Merged

Build system update: add missing ecbuild dependency JCSDA/GFDL_atmos_cubed_sphere#3

Merged

climbfuji reviewed Mar 13, 2024

View reviewed changes

climbfuji changed the title ~~Add fv3-jedi and its dependencies (SkyLab 7 version)~~ Add fv3-jedi and its dependencies (SkyLab 8 version) Apr 12, 2024

climbfuji assigned eap and srherbener Aug 8, 2024

climbfuji marked this pull request as draft June 10, 2025 20:00

climbfuji mentioned this pull request Jun 18, 2025

Spack-ify JEDI packages for FALCON (neptune-jedi) JCSDA/spack-stack#1684

Closed

7 tasks

eap unassigned srherbener Jun 24, 2025

eap moved this from Todo to Todo-nice-to-have in spack-stack-2.0.x (2025 Q4) Oct 14, 2025

climbfuji removed this from spack-stack-2.0.x (2025 Q4) Dec 5, 2025

climbfuji added this to spack-stack-2.1.x (2026 Q1) Dec 5, 2025

github-project-automation bot moved this to Todo in spack-stack-2.1.x (2026 Q1) Dec 5, 2025

climbfuji moved this from Todo to In Progress in spack-stack-2.1.x (2026 Q1) Jan 8, 2026

climbfuji mentioned this pull request Jan 22, 2026

Spackify JEDI components for NEPTUNE-JEDI JCSDA/spack-stack#1887

Merged

14 tasks

climbfuji mentioned this pull request Feb 13, 2026

CRTM: add version 3.1.3 and logic to run minimal ctests JCSDA/spack-packages#39

Merged

climbfuji closed this Feb 21, 2026

github-project-automation bot moved this from In Progress to Done in spack-stack-2.1.x (2026 Q1) Feb 21, 2026

Comments

Conversation

rhoneyager-tomorrow commented Jan 24, 2024

Description

Errata

Checklist

Uh oh!

climbfuji commented Jan 25, 2024

Uh oh!

rhoneyager-tomorrow commented Jan 25, 2024

Uh oh!

climbfuji commented Mar 11, 2024

Uh oh!

climbfuji commented Mar 11, 2024

Uh oh!

climbfuji commented Mar 11, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhoneyager-tomorrow Mar 11, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

climbfuji Mar 12, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rhoneyager-tomorrow commented Mar 11, 2024

Uh oh!

climbfuji commented Mar 11, 2024

Uh oh!

climbfuji commented Mar 11, 2024

Uh oh!

climbfuji commented Mar 12, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

climbfuji commented Mar 14, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

climbfuji commented Feb 13, 2026

Uh oh!

rickgrubin-noaa commented Feb 13, 2026

Uh oh!

climbfuji commented Feb 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

rhoneyager-tomorrow Mar 11, 2024 •

edited

Loading

climbfuji Mar 12, 2024 •

edited

Loading

climbfuji commented Mar 14, 2024 •

edited

Loading