Use IOP lat/lon in rrtmgp interface #2819

tcclevenger · 2024-05-09T19:58:44Z

Use IOP lat/lon in rrtmgp interface.

Unrelated change: dx_short code no longer needed in iop.

tcclevenger · 2024-05-09T19:59:12Z

Setting this as WIP until @bogensch is backfrom vacation.

bogensch · 2024-05-20T16:15:04Z

Thanks @tcclevenger for your work on this. I will do some testing with this branch and report back.

bogensch · 2024-05-20T22:46:00Z

I have tested this branch and it absolutely solves the problem pertaining to the incoming solar radiation.... Thus I approve of this PR and this should be merged.

Note that, unfortunately, this did not solve the bulk of the differences I am seeing in v0 and v1 DP multi-day runs (like I thought it would) and will continue to debug to track down what the source of the issue is.

tcclevenger · 2024-05-21T13:51:37Z

The hunt continues.. Thanks, @bogensch!

E3SM-Bot · 2024-05-21T14:10:46Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-05-21T14:14:19Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5452
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5730
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-05-21T14:52:11Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5452
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5730
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5452 FAILED (click to see last 100 lines of console output)


Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240521_082738_08j16c
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240521_082738_08j16c
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240521_082738_08j16c
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240521_082738_08j16c
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240521_082738_08j16c
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240521_082738_08j16c
test-scheduler took 803.883918762207 seconds'
+ errors='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240521_082738_08j16c
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240521_082738_08j16c
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240521_082738_08j16c
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240521_082738_08j16c
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240521_082738_08j16c
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240521_082738_08j16c
test-scheduler took 803.883918762207 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240521_082738_08j16c
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240521_082738_08j16c
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240521_082738_08j16c
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240521_082738_08j16c
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240521_082738_08j16c
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240521_082738_08j16c
test-scheduler took 803.883918762207 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240521_082738_08j16c
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240521_082738_08j16c
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240521_082738_08j16c
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240521_082738_08j16c
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240521_082738_08j16c
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240521_082738_08j16c
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240521_082738_08j16c
test-scheduler took 803.883918762207 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 102220 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins18302882978728660579.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5730 PASSED (click to see last 100 lines of console output)


115/132 Test #115: shoc_p3_nudging_glob_novert .............................   Passed    2.45 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_np1
116/132 Test #116: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.74 sec
        Start 117: homme_shoc_cld_p3_rrtmgp_baseline_cmp
117/132 Test #117: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.09 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_pg2_np1
118/132 Test #118: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.32 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
119/132 Test #119: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.10 sec
        Start 120: model_baseline
120/132 Test #120: model_baseline ..........................................   Passed    8.26 sec
        Start 121: model_initial
121/132 Test #121: model_initial ...........................................   Passed    4.41 sec
        Start 122: model_restart
122/132 Test #122: model_restart ...........................................   Passed    5.32 sec
        Start 123: restarted_vs_monolithic_check_np1
123/132 Test #123: restarted_vs_monolithic_check_np1 .......................   Passed    0.09 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_np1
124/132 Test #124: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    8.30 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
125/132 Test #125: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.11 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
126/132 Test #126: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    9.92 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
127/132 Test #127: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.43 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
128/132 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.55 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
129/132 Test #129: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   13.44 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
130/132 Test #130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.09 sec
        Start 131: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
131/132 Test #131: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   10.07 sec
        Start 132: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
132/132 Test #132: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.15 sec
100% tests passed, 0 tests failed out of 132
Label Time Summary:

baseline_cmp        =  87.25 secproc (16 tests)

baseline_gen        = 174.91 secproc (18 tests)

bfbhash             =   0.71 secproc (1 test)

check               =   0.70 secproc (1 test)

cld                 =  10.13 secproc (3 tests)

cld_fraction        =   2.63 secproc (1 test)

cxx baseline_cmp    =   7.62 secproc (2 tests)

diagnostics         =  42.95 secproc (22 tests)

driver              =  29.12 secproc (8 tests)

dynamics            =   5.36 secproc (3 tests)

fail                =  39.05 secproc (4 tests)

io                  =  47.23 secproc (13 tests)

mam4_optics         =   5.75 secproc (1 test)

nudging             =   9.37 secproc (2 tests)

p3                  = 119.67 secproc (7 tests)

p3_sk               = 111.96 secproc (2 tests)

physics             = 255.14 secproc (19 tests)

remap               =   4.06 secproc (1 test)

rrtmgp              =  24.98 secproc (9 tests)

shoc                =  14.04 secproc (7 tests)

spa                 =   8.95 secproc (4 tests)

surface_coupling    =   3.27 secproc (1 test)
Total Test time (real) = 662.91 sec
Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/release
Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5730/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins17245837159137221698.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

bogensch · 2024-05-29T20:49:26Z

@tcclevenger I consider this PR ready to go and should be merged in. Thanks.

E3SM-Bot · 2024-05-29T21:40:45Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-05-29T21:44:11Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5479
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5756
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: tcclevenger

E3SM-Bot · 2024-05-29T22:27:13Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5479
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5756
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	AT: RETEST;DP-SCREAM
PULLREQUESTNUM	2819
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`2faeb07`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`ae739de`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5479 FAILED (click to see last 100 lines of console output)


Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240529_155648_mglv8l
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240529_155648_mglv8l
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240529_155648_mglv8l
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240529_155648_mglv8l
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240529_155648_mglv8l
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240529_155648_mglv8l
test-scheduler took 815.2822659015656 seconds'
+ errors='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240529_155648_mglv8l
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240529_155648_mglv8l
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240529_155648_mglv8l
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240529_155648_mglv8l
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240529_155648_mglv8l
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240529_155648_mglv8l
test-scheduler took 815.2822659015656 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240529_155648_mglv8l
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240529_155648_mglv8l
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240529_155648_mglv8l
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240529_155648_mglv8l
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240529_155648_mglv8l
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240529_155648_mglv8l
test-scheduler took 815.2822659015656 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
PASS ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240529_155648_mglv8l
PASS ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240529_155648_mglv8l
PASS ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240529_155648_mglv8l
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240529_155648_mglv8l
PASS PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240529_155648_mglv8l
PASS SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240529_155648_mglv8l
PASS SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240529_155648_mglv8l
test-scheduler took 815.2822659015656 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 61403 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins17378096774083976420.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5756 PASSED (click to see last 100 lines of console output)


115/132 Test #115: shoc_p3_nudging_glob_novert .............................   Passed    2.60 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_np1
116/132 Test #116: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.51 sec
        Start 117: homme_shoc_cld_p3_rrtmgp_baseline_cmp
117/132 Test #117: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.11 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_pg2_np1
118/132 Test #118: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.19 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
119/132 Test #119: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.09 sec
        Start 120: model_baseline
120/132 Test #120: model_baseline ..........................................   Passed    8.38 sec
        Start 121: model_initial
121/132 Test #121: model_initial ...........................................   Passed    4.53 sec
        Start 122: model_restart
122/132 Test #122: model_restart ...........................................   Passed    5.52 sec
        Start 123: restarted_vs_monolithic_check_np1
123/132 Test #123: restarted_vs_monolithic_check_np1 .......................   Passed    0.10 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_np1
124/132 Test #124: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    8.20 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
125/132 Test #125: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.13 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
126/132 Test #126: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    9.73 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
127/132 Test #127: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.41 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
128/132 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.62 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
129/132 Test #129: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   13.45 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
130/132 Test #130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.09 sec
        Start 131: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
131/132 Test #131: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   10.51 sec
        Start 132: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
132/132 Test #132: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.16 sec
100% tests passed, 0 tests failed out of 132
Label Time Summary:

baseline_cmp        =  95.49 secproc (16 tests)

baseline_gen        = 185.66 secproc (18 tests)

bfbhash             =   0.71 secproc (1 test)

check               =   0.70 secproc (1 test)

cld                 =  13.43 secproc (3 tests)

cld_fraction        =   2.67 secproc (1 test)

cxx baseline_cmp    =   7.93 secproc (2 tests)

diagnostics         =  38.14 secproc (22 tests)

driver              =  28.35 secproc (8 tests)

dynamics            =   5.15 secproc (3 tests)

fail                =  51.01 secproc (4 tests)

io                  =  43.44 secproc (13 tests)

mam4_optics         =   6.12 secproc (1 test)

nudging             =  10.26 secproc (2 tests)

p3                  = 132.23 secproc (7 tests)

p3_sk               = 117.61 secproc (2 tests)

physics             = 273.58 secproc (19 tests)

remap               =   2.95 secproc (1 test)

rrtmgp              =  27.74 secproc (9 tests)

shoc                =  17.42 secproc (7 tests)

spa                 =   9.67 secproc (4 tests)

surface_coupling    =   1.56 secproc (1 test)
Total Test time (real) = 677.67 sec
Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/release
Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5756/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13026849020424016128.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

tcclevenger · 2024-05-30T13:53:40Z

@bartgol Can you take a quick look before I manual merge. DIFF in AT is expected.

Basically for rrtmgp we need to use the lat/lon from IOP file instead of the one from dynamics in the geometry data, but we don't want to overwrite geometry data for the rest of the run. So I just clone and deep copy if we have IOP object.

bartgol · 2024-05-30T14:49:12Z

@bartgol Can you take a quick look before I manual merge. DIFF in AT is expected.

Basically for rrtmgp we need to use the lat/lon from IOP file instead of the one from dynamics in the geometry data, but we don't want to overwrite geometry data for the rest of the run. So I just clone and deep copy if we have IOP object.

I wonder if it would make sense to have an iop_grid, holding just one dof. Maybe it could simplify some IO stuff. I think we should explore that route when we take on the unification of the interpolation of data from files.

tcclevenger · 2024-05-30T14:52:31Z

@bartgol Yes, I completely agree.

tcclevenger added 2 commits May 9, 2024 12:26

Remove old dx_short code

9ad56f4

Use IOP lat/lon in rrtmgp interface

2faeb07

tcclevenger added AT: WIP DP-SCREAM labels May 9, 2024

tcclevenger requested a review from bogensch May 9, 2024 19:58

tcclevenger self-assigned this May 9, 2024

bogensch approved these changes May 20, 2024

View reviewed changes

tcclevenger added AT: RETEST and removed AT: WIP labels May 21, 2024

tcclevenger requested a review from bartgol May 21, 2024 13:51

E3SM-Bot removed the AT: RETEST label May 21, 2024

tcclevenger added the AT: RETEST label May 29, 2024

E3SM-Bot removed the AT: RETEST label May 29, 2024

bartgol approved these changes May 30, 2024

View reviewed changes

tcclevenger merged commit 67fcfdd into master May 30, 2024
5 of 6 checks passed

tcclevenger deleted the tcclevenger/dp/radiation_lat_lon branch May 30, 2024 14:54

Use IOP lat/lon in rrtmgp interface #2819

Use IOP lat/lon in rrtmgp interface #2819

Conversation

tcclevenger commented May 9, 2024

tcclevenger commented May 9, 2024

bogensch commented May 20, 2024

bogensch commented May 20, 2024

tcclevenger commented May 21, 2024

E3SM-Bot commented May 21, 2024

E3SM-Bot commented May 21, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 21, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 662.91 sec

Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''full_sp_debug'''

Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''release'''

Testing '''014b212524978e3b5488b7dcb0341bbb9c5324ea''' for test '''full_debug'''

bogensch commented May 29, 2024

E3SM-Bot commented May 29, 2024

E3SM-Bot commented May 29, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented May 29, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 677.67 sec

Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''full_sp_debug'''

Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''release'''

Testing '''2d51f8af3f14c3a1d54dc1e9ef3c63e02a1e7b05''' for test '''full_debug'''

tcclevenger commented May 30, 2024

bartgol commented May 30, 2024

tcclevenger commented May 30, 2024