Enable Subfield Slices Spanning a Range of indices (Issue #1312) #2785

mjs271 · 2024-04-04T23:58:12Z

This PR adds functionality that allows a user to create a "multi-slice" subfield and gain access to the underlying view. That is, a multi-slice subfield is in contrast to the previous capability that sliced a single entry of a chosen dimension--e.g., view(:, 7, :) to use notation common to fortran/matlab, and expands this capability to allow for slicing a chosen dimension across a range of values--e.g., view(:, 2:7, :).

Summary of changes and capabilities:

Added overloads for Field::subfield() that takes the extra argument(s) index_beg, index_end to indicate the bounds of the slicing range.
Added Field::get_multi_sliced_view() that provides the view underlying the subfield as a multi-dimensional view with the same number of dimensions as the parent's associated view (as opposed to a single-sliced view that is (N - 1)-dimensional.
- Note: as implemented, this is only returned as a LayoutStride view, rather than LayoutRight.

I do need to add a few more tests and will push those changes soon, but I wanted to get this out there as a draft. Any feedback about my approach would be much-appreciated, since this is my first time doing a deep-dive into this code.

Closes #1312

mjs271 · 2024-04-10T07:22:47Z

Converting from draft to full PR after adding tests for f.subfield() and sf.get_multi_sliced_view(). Also cleaned up ekat::subview() , added tests over there, and brought the ekat submodule up to date.

Note: the ekat branch for the PR is over on my fork, since I don't yet have push access to E3SM-Project/EKAT.

E3SM-Bot · 2024-04-10T07:41:14Z

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = False

AaronDonahue · 2024-04-10T13:09:09Z

Note: the ekat branch for the PR is over on my fork, since I don't yet have push access to E3SM-Project/EKAT

@mjs271 , does this mean we will need a separate EKAT PR to accompany this one?

mahf708 · 2024-04-10T13:13:55Z

Thanks. Quick question: For future reference, how hard would it be to generalize this to take noncontiguous slices? Say by passing a sorted/unsorted map/list?

@AaronDonahue there's a PR in EKAT upstream, but probably a clean EKAT submodule update is in order? E3SM-Project/EKAT#333

mjs271 · 2024-04-10T16:12:17Z

@AaronDonahue @mahf708
That's correct--I've got a PR open on ekat, but I'm not a member of that repo, so this PR has the submodule commit pointed at my personal fork. Happy to resolve that however seems best

And, @mahf708, in response to your question, I've given it a little thought, and I don't think it'd be an enormous undertaking, but I do think it would require a new approach. The reason being that, as far as I know, kokkos allows slicing only with scalars, {std,Kokkos}::pair<>(), and Kokkos::ALL. As I was working on this code, my thought for the noncontiguous slicing approach was that it'd require taking individual contiguous slices and then concatenating. However, there's a nonzero chance there's another clever approach I'm not thinking of

It also occurred to me that it could be nice to be able to slice multiple dimensions at once, but I'm not sure whether there's a ready use case and also whether that might be approaching too much complexity

E3SM-Bot · 2024-04-10T22:20:08Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5173
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`16d63ec`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5521
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`16d63ec`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

mahf708 · 2024-04-10T22:33:40Z

Thanks @mjs271!

As I was working on this code, my thought for the noncontiguous slicing approach was that it'd require taking individual contiguous slices and then concatenating.

Yep, that's what I was thinking too. We can circle back to it later when there's more of a need along with multi-dim slicing 😸

E3SM-Bot · 2024-04-10T23:08:07Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5173
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`16d63ec`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5521
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`16d63ec`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5173 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.114.3' to the list of known hosts.

fatal: reference is not a tree: 856f699d531faddd5a3626d62cc9992c0a8efdd0

Unable to checkout '856f699d531faddd5a3626d62cc9992c0a8efdd0' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1003)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)

at jdk.internal.reflect.GeneratedMethodAccessor232.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)

at jdk.proxy31/jdk.proxy31.$Proxy101.execute(Unknown Source)

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

Caused: hudson.plugins.git.GitException

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:377)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused: java.io.IOException: Could not perform submodule update

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 73338 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins13237062574134617540.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5521 PASSED (click to see last 100 lines of console output)


        Start 105: shoc_p3_nudged
105/122 Test #105: shoc_p3_nudged ..........................................   Passed    1.77 sec
        Start 106: shoc_p3_nudged_remapped
106/122 Test #106: shoc_p3_nudged_remapped .................................   Passed    1.96 sec
        Start 107: shoc_p3_nudging_glob_novert
107/122 Test #107: shoc_p3_nudging_glob_novert .............................   Passed    1.84 sec
        Start 108: homme_shoc_cld_p3_rrtmgp_np1
108/122 Test #108: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    4.27 sec
        Start 109: homme_shoc_cld_p3_rrtmgp_baseline_cmp
109/122 Test #109: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.15 sec
        Start 110: homme_shoc_cld_p3_rrtmgp_pg2_np1
110/122 Test #110: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    4.15 sec
        Start 111: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
111/122 Test #111: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.13 sec
        Start 112: model_baseline
112/122 Test #112: model_baseline ..........................................   Passed    5.23 sec
        Start 113: model_initial
113/122 Test #113: model_initial ...........................................   Passed    5.49 sec
        Start 114: model_restart
114/122 Test #114: model_restart ...........................................   Passed    4.82 sec
        Start 115: restarted_vs_monolithic_check_np1
115/122 Test #115: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 116: homme_shoc_cld_spa_p3_rrtmgp_np1
116/122 Test #116: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    4.94 sec
        Start 117: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
117/122 Test #117: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.23 sec
        Start 118: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
118/122 Test #118: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    5.77 sec
        Start 119: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
119/122 Test #119: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.15 sec
        Start 120: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
120/122 Test #120: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    1.17 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
121/122 Test #121: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   17.83 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
122/122 Test #122: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.23 sec
100% tests passed, 0 tests failed out of 122
Label Time Summary:

baseline_cmp        =  91.85 secproc (14 tests)

baseline_gen        = 154.54 secproc (16 tests)

bfbhash             =   0.88 secproc (1 test)

check               =   0.85 secproc (1 test)

cld                 =   8.72 secproc (3 tests)

cld_fraction        =   1.84 secproc (1 test)

cxx baseline_cmp    =  10.85 secproc (2 tests)

diagnostics         =  30.72 secproc (18 tests)

driver              =  18.06 secproc (7 tests)

dynamics            =   5.57 secproc (3 tests)

fail                =  53.03 secproc (4 tests)

io                  =  37.48 secproc (12 tests)

nudging             =   7.09 secproc (2 tests)

p3                  =  98.73 secproc (7 tests)

p3_sk               = 107.63 secproc (2 tests)

physics             = 221.90 secproc (18 tests)

remap               =   3.19 secproc (1 test)

rrtmgp              =  16.87 secproc (8 tests)

shoc                =  12.27 secproc (7 tests)

spa                 =   9.17 secproc (4 tests)

surface_coupling    =   1.89 sec*proc (1 test)
Total Test time (real) = 779.67 sec
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/release
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5521/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins12955974405537412289.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

E3SM-Bot · 2024-04-11T12:39:27Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5179
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5523
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-04-11T13:29:59Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5179
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5523
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5179 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.113.3' to the list of known hosts.

fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d

Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1003)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)

at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)

at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

Caused: hudson.plugins.git.GitException

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:377)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused: java.io.IOException: Could not perform submodule update

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 15198 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5730545843742489020.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5523 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    1.98 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    6.01 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.20 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.40 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.18 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    7.01 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.22 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.81 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.08 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    7.94 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.21 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.62 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.16 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.86 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.11 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.26 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed    9.61 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.19 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:

baseline_cmp        =  60.47 secproc (16 tests)

baseline_gen        = 145.91 secproc (18 tests)

bfbhash             =   0.86 secproc (1 test)

check               =   0.77 secproc (1 test)

cld                 =   9.60 secproc (3 tests)

cld_fraction        =   2.33 secproc (1 test)

cxx baseline_cmp    =   8.40 secproc (2 tests)

diagnostics         =  37.05 secproc (19 tests)

driver              =  24.66 secproc (8 tests)

dynamics            =   5.23 secproc (3 tests)

fail                =  40.14 secproc (4 tests)

io                  =  36.30 secproc (12 tests)

mam4_optics         =   5.70 secproc (1 test)

nudging             =   8.27 secproc (2 tests)

p3                  = 135.47 secproc (7 tests)

p3_sk               =  41.88 secproc (2 tests)

physics             = 198.59 secproc (19 tests)

remap               =   3.20 secproc (1 test)

rrtmgp              =  22.60 secproc (9 tests)

shoc                =  13.12 secproc (7 tests)

spa                 =   9.60 secproc (4 tests)

surface_coupling    =   2.28 secproc (1 test)
Total Test time (real) = 582.95 sec
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/release
Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5523/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins3502710905521686397.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

E3SM-Bot · 2024-04-11T15:46:25Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5182
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5526
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-04-11T16:51:58Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5182
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5526
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5182 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.113.4' to the list of known hosts.

fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d

Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1003)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)

at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)

at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

Caused: hudson.plugins.git.GitException

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:377)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused: java.io.IOException: Could not perform submodule update

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 123157 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins11852718006915602681.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5526 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    2.74 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    6.64 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.18 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    6.99 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.11 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    7.51 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.23 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.79 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.10 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    6.29 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.16 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.77 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.17 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    1.01 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.06 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.27 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed    7.38 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.20 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:

baseline_cmp        =  59.37 secproc (16 tests)

baseline_gen        = 144.95 secproc (18 tests)

bfbhash             =   0.79 secproc (1 test)

check               =   0.72 secproc (1 test)

cld                 =  11.86 secproc (3 tests)

cld_fraction        =   2.14 secproc (1 test)

cxx baseline_cmp    =   9.28 secproc (2 tests)

diagnostics         =  38.14 secproc (19 tests)

driver              =  29.65 secproc (8 tests)

dynamics            =   5.15 secproc (3 tests)

fail                =  40.33 secproc (4 tests)

io                  =  42.27 secproc (12 tests)

mam4_optics         =   5.18 secproc (1 test)

nudging             =   9.62 secproc (2 tests)

p3                  = 131.79 secproc (7 tests)

p3_sk               =  97.02 secproc (2 tests)

physics             = 251.34 secproc (19 tests)

remap               =   4.59 secproc (1 test)

rrtmgp              =  25.72 secproc (9 tests)

shoc                =  15.75 secproc (7 tests)

spa                 =  10.61 secproc (4 tests)

surface_coupling    =   4.35 secproc (1 test)
Total Test time (real) = 647.87 sec
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/release
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5526/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins15109751008381081617.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

E3SM-Bot · 2024-04-11T20:36:03Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5186
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5530
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-04-11T21:27:51Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5186
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5530
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`52500ae`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5186 FAILED (click to see last 100 lines of console output)


Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive cime # timeout=10
using GIT_SSH to set credentials jgfouca github key
Verifying host key using manually-configured host key entries
 > git submodule update --init --recursive externals/ekat # timeout=10
hudson.plugins.git.GitException: Command "git submodule update --init --recursive externals/ekat" returned status code 1:
stdout: Cloning into 'externals/ekat'...
stderr: Warning: Permanently added the ECDSA host key for IP address '140.82.114.3' to the list of known hosts.

fatal: reference is not a tree: 1862414b96d023b53a36c5d5dcc968b248b60d0d

Unable to checkout '1862414b96d023b53a36c5d5dcc968b248b60d0d' in submodule path 'externals/ekat'
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2846)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2185)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1573)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
at java.base/java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:184)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:77)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:70)

Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to mappy

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1787)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1003)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)

at jdk.internal.reflect.GeneratedMethodAccessor153.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:568)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)

at jdk.proxy29/jdk.proxy29.$Proxy158.execute(Unknown Source)

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:196)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

Caused: hudson.plugins.git.GitException

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:89)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:69)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:47)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1576)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:377)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)

at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

at java.base/java.lang.Thread.run(Thread.java:829)

Caused: java.io.IOException: Could not perform submodule update

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:201)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1390)

at hudson.scm.SCM.checkout(SCM.java:540)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1248)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:649)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:521)

at hudson.model.Run.execute(Run.java:1895)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:44)

at hudson.model.ResourceController.execute(ResourceController.java:101)

at hudson.model.Executor.run(Executor.java:442)

$ ssh-agent -k

unset SSH_AUTH_SOCK;

unset SSH_AGENT_PID;

echo Agent pid 119445 killed;

[ssh-agent] Stopped.

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins316686276118700470.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5530 PASSED (click to see last 100 lines of console output)


        Start 112: shoc_p3_nudging_glob_novert
112/129 Test #112: shoc_p3_nudging_glob_novert .............................   Passed    1.90 sec
        Start 113: homme_shoc_cld_p3_rrtmgp_np1
113/129 Test #113: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    7.58 sec
        Start 114: homme_shoc_cld_p3_rrtmgp_baseline_cmp
114/129 Test #114: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.15 sec
        Start 115: homme_shoc_cld_p3_rrtmgp_pg2_np1
115/129 Test #115: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    7.44 sec
        Start 116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
116/129 Test #116: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.13 sec
        Start 117: model_baseline
117/129 Test #117: model_baseline ..........................................   Passed    8.39 sec
        Start 118: model_initial
118/129 Test #118: model_initial ...........................................   Passed    4.24 sec
        Start 119: model_restart
119/129 Test #119: model_restart ...........................................   Passed    4.81 sec
        Start 120: restarted_vs_monolithic_check_np1
120/129 Test #120: restarted_vs_monolithic_check_np1 .......................   Passed    0.09 sec
        Start 121: homme_shoc_cld_spa_p3_rrtmgp_np1
121/129 Test #121: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    8.10 sec
        Start 122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
122/129 Test #122: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.22 sec
        Start 123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
123/129 Test #123: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    6.80 sec
        Start 124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
124/129 Test #124: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.17 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
125/129 Test #125: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.87 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1
126/129 Test #126: homme_shoc_cld_spa_p3_rrtmgp_dp_np1 .....................   Passed   20.02 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp
127/129 Test #127: homme_shoc_cld_spa_p3_rrtmgp_dp_baseline_cmp ............   Passed    0.21 sec
        Start 128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
128/129 Test #128: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   10.32 sec
        Start 129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
129/129 Test #129: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.17 sec
100% tests passed, 0 tests failed out of 129
Label Time Summary:

baseline_cmp        =  86.20 secproc (16 tests)

baseline_gen        = 178.18 secproc (18 tests)

bfbhash             =   0.88 secproc (1 test)

check               =   0.80 secproc (1 test)

cld                 =  10.57 secproc (3 tests)

cld_fraction        =   2.29 secproc (1 test)

cxx baseline_cmp    =   7.13 secproc (2 tests)

diagnostics         =  46.00 secproc (19 tests)

driver              =  30.24 secproc (8 tests)

dynamics            =   5.08 secproc (3 tests)

fail                =  48.93 secproc (4 tests)

io                  =  42.13 secproc (12 tests)

mam4_optics         =   5.82 secproc (1 test)

nudging             =  10.17 secproc (2 tests)

p3                  = 142.07 secproc (7 tests)

p3_sk               = 123.55 secproc (2 tests)

physics             = 292.73 secproc (19 tests)

remap               =   5.99 secproc (1 test)

rrtmgp              =  28.01 secproc (9 tests)

shoc                =  14.65 secproc (7 tests)

spa                 =   9.60 secproc (4 tests)

surface_coupling    =   3.73 secproc (1 test)
Total Test time (real) = 711.15 sec
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/release
Testing '''a32dba4fdd314a7622254a344dcb94e1cb631ca7''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5530/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins720872902790362728.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

E3SM-Bot · 2024-04-11T23:56:55Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5190
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5534
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-04-12T00:27:47Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5190
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5534
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5190 FAILED (click to see last 100 lines of console output)


    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ errors='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM STANDALONE TESTING FAILED!
Build type full_debug failed at build time. Here's the build log:
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx && module purge && module load sems-archive-env acme-env acme-cmake/3.26.3 acme-gcc/11.2.0 sems-archive-git/2.10.1 acme-openmpi/4.1.4 acme-netcdf/4.7.4/acme && export GATOR_INITIAL_MB=4000MB && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m mappy
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/5190/scream/components/eamxx
mappy failed
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
DIFF ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240411_180348_9eaeke
DIFF ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240411_180348_9eaeke
DIFF ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240411_180348_9eaeke
DIFF ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240411_180348_9eaeke
DIFF PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240411_180348_9eaeke
DIFF SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240411_180348_9eaeke
DIFF SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (phase BASELINE)
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240411_180348_9eaeke
test-scheduler took 796.1405007839203 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 20978 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins3388199214036993032.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5534 FAILED (click to see last 100 lines of console output)


  534 |                         pg%g_lat, pg%g_dofs_per_rank, pg%g_dofs_offsets, MPIreal_t, &
      |                        1
......
  593 |             dofs_per_elem, g_elem_per_rank, g_elem_offsets, MPIinteger_t, par%comm, ierr)
      |            2            
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(8)/INTEGER(4)).
[ 75%] Linking CXX executable io_diags
[ 75%] Built target property_checks
[ 75%] Building Fortran object src/dynamics/homme/tests/CMakeFiles/scream_theta-l_kokkos_4_72_10.dir/__/interface/homme_grid_mod.F90.o
[ 75%] Built target io_se_grid
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp: In function 'scream::p3::unit_test::UnitWrap::UnitTest >::TestUpwind::run_phys()::':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:190:22: warning: 'r_min0' may be used uninitialized [-Wmaybe-uninitialized]
  190 |           if (r_min1 < r_min0 - 10*eps) ++nerr;
      |              ~~~~~~~~^~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:150:15: note: 'r_min0' was declared here
  150 |           Scalar mass0, r_min0, r_max0;
      |               ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:190:1: warning: 'r_min1' may be used uninitialized [-Wmaybe-uninitialized]
  190 |           if (r_min1 < r_min0 - 10*eps) ++nerr;
      | ^ 
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:180:15: note: 'r_min1' was declared here
  180 |           Scalar mass1, r_min1, r_max1;
      |               ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:191:22: warning: 'r_max0' may be used uninitialized [-Wmaybe-uninitialized]
  191 |           if (r_max1 > r_max0 + 10*eps) ++nerr;
      |              ~~~~~~~~^~~~~~~~~~~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:150:23: note: 'r_max0' was declared here
  150 |           Scalar mass0, r_min0, r_max0;
      |                       ^~~~~~
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:191:1: warning: 'r_max1' may be used uninitialized [-Wmaybe-uninitialized]
  191 |           if (r_max1 > r_max0 + 10*eps) ++nerr;
      | ^ 
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_upwind_unit_tests.cpp:180:23: note: 'r_max1' was declared here
  180 |           Scalar mass1, r_min1, r_max1;
      |                       ^~~~~~
[ 75%] Built target io_filled
nvcc_internal_extended_lambda_implementation: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::p3::unit_test::UnitWrap::UnitTest >::TestUpwind::run_phys()::; Tag = __nv_dl_tag >::TestUpwind::run_phys, 2>; OpFuncR = void; OpFuncArgs = {const Kokkos::Impl::CudaTeamMember&, int&}; F1 = Kokkos::Array*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >, 2, void>; F2 = Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F3 = Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<0> >; F4 = int; F5 = const double; F6 = const int; F7 = const int; F8 = int; F9 = const Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >; F10 = const Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >; F11 = const Kokkos::View*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >; F12 = int; F13 = const double; F14 = Kokkos::Array*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >, 2, void>; F15 = Kokkos::Array*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >, 2, void>; F16 = Kokkos::Array*, Kokkos::LayoutRight, Kokkos::Device, Kokkos::MemoryTraits<1> >, 2, void>; F17 = const Kokkos::View, Kokkos::MemoryTraits<1> >]':
nvcc_internal_extended_lambda_implementation:379:19: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
[ 75%] Built target io_diags
[ 75%] Linking CXX executable output_restart
[ 75%] Linking CXX executable field_utils
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/share/field/field_alloc_prop.hpp: In copy constructor 'scream::FieldAllocProp::FieldAllocProp(const scream::FieldAllocProp&)':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/share/field/field_alloc_prop.hpp:105:1: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  105 |   FieldAllocProp (const FieldAllocProp&) = default;
      | ^ ~~~~~~~~~~~~
[ 75%] Built target output_restart
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h: In instantiation of 'constexpr std::pair::type>::__type, typename std::__strip_reference_wrapper::type>::__type> std::make_pair(_T1&&, _T2&&) [with _T1 = double; _T2 = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_Tp2>::type = double; typename std::__strip_reference_wrapper::type>::__type = double; typename std::decay<_Tp>::type = double]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:204:37:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb_homogeneous_freezing() [with D = Kokkos::Device]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:253:31:   required from 'static void scream::p3::unit_test::UnitWrap::UnitTest::TestIceSed::run_bfb() [with D = Kokkos::Device]'
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_sed_unit_tests.cpp:269:6:   required from here
/projects/ppc64le-pwr9-rhel8/compilers/gcc/11.3.0/gcc/8.3.1/base/tchbki3/lib/gcc/powerpc64le-unknown-linux-gnu/11.3.0/../../../../include/c++/11.3.0/bits/stl_pair.h:567:1: note: parameter passing for argument of type 'std::pair' when C++17 is enabled changed to match C++14 in GCC 10.1
  567 |     make_pair(_T1&& __x, _T2&& __y)
      | ^   ~~~~~
[ 75%] Built target field_utils
[ 76%] Linking CXX executable time_interpolation
[ 76%] Linking CXX executable atm_proc
[ 76%] Built target time_interpolation
[ 76%] Built target atm_proc
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3CloudWaterAutoconversion::run_physics, 1>, void(const int&, int&)>; ReducerType = Kokkos::InvalidType; Traits = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Range.hpp:321:6: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
  321 |   inline void execute() {
      |      ^~~~~~~
[ 76%] Linking CXX static library libscream_rrtmgp.a
[ 76%] Built target scream_rrtmgp
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp: In static member function 'static void* __nv_hdl_wrapper_t::manager::do_copy(void*) [with Lambda = scream::p3::unit_test::UnitWrap::UnitTest >::TestIceNucleation::run_ice_nucleation_bfb()::; Tag = __nv_dl_tag >::TestIceNucleation::run_ice_nucleation_bfb, 1>; OpFuncR = void; OpFuncArgs = {const int&}; F1 = Kokkos::View, Kokkos::MemoryTraits<0> >; F2 = bool; F3 = bool]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx/src/physics/p3/tests/p3_ice_nucleation_unit_tests.cpp:69:348: note: the layout of aggregates containing vectors with 2-byte alignment has changed in GCC 5
   69 |         Kokkos::parallel_for(num_test_itrs, KOKKOS_LAMBDA(const Int& i) {
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp: In member function 'void Kokkos::Impl::ParallelReduce, ReducerType, Kokkos::Cuda>::execute() [with FunctorType = __nv_hdl_wrapper_t >::TestP3SubgridVarianceScaling::run_property_tests, 1>, void(const Kokkos::Impl::CudaTeamMember&, int&)>; ReducerType = Kokkos::InvalidType; Properties = {Kokkos::Cuda}]':
/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/externals/ekat/extern/kokkos/core/src/Cuda/Kokkos_Cuda_Parallel_Team.hpp:839:6: note: the layout of aggregates containing vectors with 8-byte alignment has changed in GCC 5
  839 |   inline void execute() {
      |      ^~~~~~~
[ 76%] Linking CXX executable field
[ 76%] Built target field
[ 76%] Linking CXX static library libscream_theta-l_kokkos_4_72_10.a
[ 76%] Built target scream_theta-l_kokkos_4_72_10
[ 76%] Linking CXX executable p3_tests
[ 76%] Built target p3_tests
gmake: *** [Makefile:166: all] Error 2
Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5534/scream/components/eamxx

weaver failed

######################################################

Build step 'Execute shell' marked build as failure

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins13985744753253938770.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

E3SM-Bot · 2024-04-12T11:33:56Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5197
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5536
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`f5bfe88`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`a6c73f4`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-05-31T15:23:18Z

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

E3SM-Bot · 2024-05-31T15:23:25Z

All Jobs Finished; status = PASSED, target_sha=d0adfce2ba20d75dc008be3510b28275b5c0f0bd, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-01T15:39:58Z

All Jobs Finished; status = PASSED, target_sha=d0adfce2ba20d75dc008be3510b28275b5c0f0bd, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-02T15:58:02Z

All Jobs Finished; status = PASSED, target_sha=d0adfce2ba20d75dc008be3510b28275b5c0f0bd, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-03T16:10:00Z

All Jobs Finished; status = PASSED, target_sha=d0adfce2ba20d75dc008be3510b28275b5c0f0bd, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-03T19:44:04Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: d0adfce
current base branch sha : 9f8ce1f
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-06-03T19:45:44Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5501
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5777
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-06-03T20:42:59Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5501
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5777
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-06-03T20:43:16Z

All Jobs Finished; status = PASSED, target_sha=9f8ce1fa49acc5793cdee7c3c5fae94e979025e8, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-04T20:03:13Z

The base branch has been updated since the last successful testing.

last PASS base branch sha: 9f8ce1f
current base branch sha : b80e1bd
The AutoTester will discard the last PASS, and re-test the PR from scratch

E3SM-Bot · 2024-06-04T20:05:11Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5504
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5780
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-06-04T20:59:29Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5504
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5780
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

E3SM-Bot · 2024-06-04T20:59:46Z

All Jobs Finished; status = PASSED, target_sha=b80e1bdc3f2e4e66d2f8c2a2f97a395cb34c8fd6, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-05T21:10:09Z

All Jobs Finished; status = PASSED, target_sha=b80e1bdc3f2e4e66d2f8c2a2f97a395cb34c8fd6, However Inspection must be performed before merge can occur...

E3SM-Bot · 2024-06-07T19:14:04Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-06-08T22:05:35Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-06-09T23:10:47Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-06-11T03:44:32Z

Status Flag 'Pull Request AutoTester' - User Requested Retest - Label AT: RETEST will be reset after testing.

E3SM-Bot · 2024-06-11T22:01:01Z

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5520
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;AT: RETEST;AT: AUTOMERGE;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5791
Status: STARTED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;AT: RETEST;AT: AUTOMERGE;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)

Pull Request Author: mjs271

E3SM-Bot · 2024-06-11T22:42:39Z

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 2 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Build Num: 5520
Status: FAILED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;AT: RETEST;AT: AUTOMERGE;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Build Num: 5791
Status: PASSED

Jenkins Parameters

Parameter Name	Value
PR_LABELS	enhancement;AT: RETEST;AT: AUTOMERGE;EKAT;code usability;Quality of Life
PULLREQUESTNUM	2785
SCREAM_SOURCE_REPO	https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA	`b417968`
SCREAM_TARGET_BRANCH	master
SCREAM_TARGET_REPO	https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA	`6557feb`
TEST_REPO_ALIAS	SCREAM

SCREAM_PullRequest_Autotester_Mappy # 5520 FAILED (click to see last 100 lines of console output)


Waiting for tests to finish
NLFAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240611_144017_64fber
NLFAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240611_144017_64fber
NLFAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240611_144017_64fber
NLFAIL ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240611_144017_64fber
NLFAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240611_144017_64fber
NLFAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240611_144017_64fber
test-scheduler took 890.7051057815552 seconds'
+ errors='Waiting for tests to finish
NLFAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240611_144017_64fber
NLFAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240611_144017_64fber
NLFAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240611_144017_64fber
NLFAIL ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240611_144017_64fber
NLFAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240611_144017_64fber
NLFAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240611_144017_64fber
test-scheduler took 890.7051057815552 seconds'
+ V1_FAILURES_DETAILS+='Waiting for tests to finish
NLFAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240611_144017_64fber
NLFAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240611_144017_64fber
NLFAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240611_144017_64fber
NLFAIL ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240611_144017_64fber
NLFAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240611_144017_64fber
NLFAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240611_144017_64fber
test-scheduler took 890.7051057815552 seconds'
+ set +x
######################################################
FAILS DETECTED:
  SCREAM V1 TESTING FAILED!
Waiting for tests to finish
NLFAIL ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_D_Lh4.ne4_ne4.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERP_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-4.C.20240611_144017_64fber
NLFAIL ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_D_Ln22.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-rad_frequency_2--scream-output-preset-5.C.20240611_144017_64fber
NLFAIL ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_Ln9.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu.scream-output-preset-2.C.20240611_144017_64fber
NLFAIL ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/ERS_P16_Ln22.ne30_ne30.F2010-SCREAMv1-DP-DYCOMSrf01.mappy_gnu.C.20240611_144017_64fber
NLFAIL PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/PET_Ln9_P32x2.ne4pg2_ne4pg2.F2010-SCREAMv1.mappy_gnu.scream-output-preset-1.C.20240611_144017_64fber
NLFAIL SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln5.ne4pg2_oQU480.F2010-SCREAMv1-MPASSI.mappy_gnu.scream-mam4xx-optics.C.20240611_144017_64fber
NLFAIL SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3 (but otherwise OK) RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln9.ne4_ne4.F2010-SCREAMv1-noAero.mappy_gnu.scream-output-preset-3.C.20240611_144017_64fber
test-scheduler took 890.7051057815552 seconds
######################################################
Build step 'Execute shell' marked build as failure
$ ssh-agent -k
unset SSH_AUTH_SOCK;
unset SSH_AGENT_PID;
echo Agent pid 25663 killed;
[ssh-agent] Stopped.
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le
cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh
We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs
squeue -o"%.7i %u %40j" | grep e3sm-jenkins | grep test-launcher | awk '{ print $1 }' | xargs -r scancel
[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins16157243609859373541.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

SCREAM_PullRequest_Autotester_Weaver # 5791 PASSED (click to see last 100 lines of console output)


116/133 Test #116: shoc_p3_nudging_glob_novert .............................   Passed    2.52 sec
        Start 117: homme_shoc_cld_p3_rrtmgp_np1
117/133 Test #117: homme_shoc_cld_p3_rrtmgp_np1 ............................   Passed    6.57 sec
        Start 118: homme_shoc_cld_p3_rrtmgp_baseline_cmp
118/133 Test #118: homme_shoc_cld_p3_rrtmgp_baseline_cmp ...................   Passed    0.12 sec
        Start 119: homme_shoc_cld_p3_rrtmgp_pg2_np1
119/133 Test #119: homme_shoc_cld_p3_rrtmgp_pg2_np1 ........................   Passed    6.58 sec
        Start 120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp
120/133 Test #120: homme_shoc_cld_p3_rrtmgp_pg2_baseline_cmp ...............   Passed    0.09 sec
        Start 121: model_baseline
121/133 Test #121: model_baseline ..........................................   Passed    7.81 sec
        Start 122: model_initial
122/133 Test #122: model_initial ...........................................   Passed    4.65 sec
        Start 123: model_restart
123/133 Test #123: model_restart ...........................................   Passed    5.67 sec
        Start 124: restarted_vs_monolithic_check_np1
124/133 Test #124: restarted_vs_monolithic_check_np1 .......................   Passed    0.13 sec
        Start 125: homme_shoc_cld_spa_p3_rrtmgp_np1
125/133 Test #125: homme_shoc_cld_spa_p3_rrtmgp_np1 ........................   Passed    7.55 sec
        Start 126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp
126/133 Test #126: homme_shoc_cld_spa_p3_rrtmgp_baseline_cmp ...............   Passed    0.13 sec
        Start 127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1
127/133 Test #127: homme_shoc_cld_spa_p3_rrtmgp_128levels_np1 ..............   Passed    9.40 sec
        Start 128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1
128/133 Test #128: homme_shoc_cld_spa_p3_rrtmgp_128levels_tend_check_np1 ...   Passed    1.35 sec
        Start 129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp
129/133 Test #129: homme_shoc_cld_spa_p3_rrtmgp_128levels_baseline_cmp .....   Passed    0.69 sec
        Start 130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1
130/133 Test #130: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_np1 .................   Passed   13.95 sec
        Start 131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp
131/133 Test #131: homme_shoc_cld_spa_p3_rrtmgp_pg2_dp_baseline_cmp ........   Passed    0.09 sec
        Start 132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1
132/133 Test #132: homme_shoc_cld_p3_mam_optics_rrtmgp_np1 .................   Passed   10.66 sec
        Start 133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp
133/133 Test #133: homme_shoc_cld_p3_mam_optics_rrtmgp_baseline_cmp ........   Passed    0.18 sec
100% tests passed, 0 tests failed out of 133
Label Time Summary:

baseline_cmp        =  94.90 secproc (16 tests)

baseline_gen        = 175.00 secproc (18 tests)

bfbhash             =   0.70 secproc (1 test)

check               =   0.70 secproc (1 test)

cld                 =  12.00 secproc (3 tests)

cld_fraction        =   0.86 secproc (1 test)

cxx baseline_cmp    =   7.81 secproc (2 tests)

diagnostics         =  29.33 secproc (22 tests)

driver              =  27.00 secproc (8 tests)

dynamics            =   6.12 secproc (3 tests)

fail                =  47.54 secproc (4 tests)

io                  =  43.21 secproc (13 tests)

mam4_optics         =   5.76 secproc (1 test)

nudging             =   8.96 secproc (2 tests)

p3                  = 142.71 secproc (7 tests)

p3_sk               = 123.03 secproc (2 tests)

physics             = 285.32 secproc (19 tests)

remap               =   2.60 secproc (1 test)

rrtmgp              =  24.89 secproc (9 tests)

shoc                =  15.71 secproc (7 tests)

spa                 =   8.82 secproc (4 tests)

surface_coupling    =   3.34 secproc (1 test)
Total Test time (real) = 677.52 sec
Testing '''d2687709c0a6b47c85f81d4ba94b9ae148d67de0''' for test '''full_sp_debug'''
RUN: taskset -c 52-103 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_sp_debug
Testing '''d2687709c0a6b47c85f81d4ba94b9ae148d67de0''' for test '''release'''
RUN: taskset -c 104-155 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/release
Testing '''d2687709c0a6b47c85f81d4ba94b9ae148d67de0''' for test '''full_debug'''
RUN: taskset -c 0-51 sh -c '''SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/ctest_script.cmake -DCTEST_SITE=weaver -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-fortran/4.6.0/gcc/11.3.0/openmpi/4.1.4/5ka6asw -DNetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/netcdf-c/4.9.0/gcc/11.3.0/openmpi/4.1.4/mdd6fth -DPnetCDF_C_PATH=/projects/ppc64le-pwr9-rhel8/tpls/parallel-netcdf/1.12.3/gcc/11.3.0/openmpi/4.1.4/52dibdr -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='''''''''ON''''''''' -DCMAKE_CXX_COMPILER=mpicxx -DCMAKE_C_COMPILER=mpicc -DCMAKE_Fortran_COMPILER=mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_BASELINES_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug" '''

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx && source /etc/profile.d/modules.sh && module purge && module load cmake/3.25.1 git/2.39.1 python/3.10.8 py-netcdf4/1.5.8 gcc/11.3.0 cuda/11.8.0 openmpi netcdf-c netcdf-fortran parallel-netcdf netlib-lapack && export HDF5_USE_FILE_LOCKING=FALSE && true && bsub -I -q rhel8 -n 4 -gpu num=4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -p -c EKAT_DISABLE_TPL_WARNINGS=ON -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/5791/scream/components/eamxx

Completed analysis on weaver'

[[ 0 != 0 ]]
[[ 1 == 0 ]]
[[ weaver == \m\a\p\p\y ]]
set +x

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/
./scream/components/eamxx/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins3364101615612103891.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: SUCCESS

bartgol · 2024-06-11T23:01:53Z

NL fails are due to the upstream merge (we have to bless). Merging.

mjs271 requested review from bartgol, jgfouca and AaronDonahue April 4, 2024 23:58

mjs271 mentioned this pull request Apr 5, 2024

Add get_subview() Implementations for Multi-Sliced Subfields E3SM-Project/EKAT#333

Merged

mjs271 requested a review from tcclevenger April 5, 2024 00:17

mjs271 marked this pull request as ready for review April 10, 2024 07:19

mjs271 added enhancement New feature or request EKAT code usability Quality of Life labels Apr 10, 2024

mjs271 self-assigned this Apr 10, 2024

mahf708 changed the title ~~Enable Subfield Slices Spanning a Range of indices (Issue #1312)--DRAFT~~ Enable Subfield Slices Spanning a Range of indices (Issue #1312) Apr 10, 2024

mjs271 force-pushed the mjs/eamxx/fix-#1312 branch from c3a3ed2 to 16d63ec Compare April 10, 2024 18:32

jgfouca approved these changes Jun 6, 2024

View reviewed changes

jgfouca requested a review from bartgol June 6, 2024 18:54

jgfouca added the AT: AUTOMERGE label Jun 6, 2024

AaronDonahue added the AT: RETEST label Jun 7, 2024

bartgol approved these changes Jun 10, 2024

View reviewed changes

E3SM-Bot removed the AT: RETEST label Jun 11, 2024

bartgol merged commit a15130c into master Jun 11, 2024
7 of 8 checks passed

bartgol deleted the mjs/eamxx/fix-#1312 branch June 11, 2024 23:01

bartgol mentioned this pull request Dec 6, 2024

Fix alloc size in FieldAllocProp #2844

Closed

Enable Subfield Slices Spanning a Range of indices (Issue #1312) #2785

Enable Subfield Slices Spanning a Range of indices (Issue #1312) #2785

Conversation

mjs271 commented Apr 4, 2024

mjs271 commented Apr 10, 2024 • edited Loading

E3SM-Bot commented Apr 10, 2024

AaronDonahue commented Apr 10, 2024

mahf708 commented Apr 10, 2024

mjs271 commented Apr 10, 2024 • edited Loading

E3SM-Bot commented Apr 10, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

mahf708 commented Apr 10, 2024

E3SM-Bot commented Apr 10, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 779.67 sec

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''release'''

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_sp_debug'''

Testing '''16d63ec1f1e2a1d356d2faf4d8bdd8cf3fbbe560''' for test '''full_debug'''

E3SM-Bot commented Apr 11, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented Apr 11, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 582.95 sec

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_sp_debug'''

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''release'''

Testing '''d9c0bc846ffc60d14b68586182183bfa9a3c359d''' for test '''full_debug'''

E3SM-Bot commented Apr 11, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented Apr 11, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

We're having issues with some test-launcher job hanging forever. So let's make sure we clean all penting test-launcher jobs

Total Test time (real) = 647.87 sec

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''release'''

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_sp_debug'''

Testing '''50f995e8afe99f8bbef9242a1f3806095ed19f17''' for test '''full_debug'''

E3SM-Bot commented Apr 11, 2024

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

Jenkins Parameters

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

Jenkins Parameters

Using Repos:

E3SM-Bot commented Apr 11, 2024

Build Information

mjs271 commented Apr 10, 2024 •

edited

Loading

mjs271 commented Apr 10, 2024 •

edited

Loading