Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add logging support in SCREAM #1520

Merged
merged 15 commits into from
Apr 8, 2022
Merged

Add logging support in SCREAM #1520

merged 15 commits into from
Apr 8, 2022

Conversation

bartgol
Copy link
Contributor

@bartgol bartgol commented Apr 1, 2022 •
edited
Loading

This PR adds support for logging to screen/file in SCREAM. For CIME runs, it ensures that the log ends up in the atm.log file.

Closes #1502.

@bartgol bartgol added enhancement New feature or request AT: WIP SCREAMv1 labels Apr 1, 2022
@bartgol bartgol self-assigned this Apr 1, 2022
@bartgol bartgol force-pushed the bartgol/logging-support branch from feee089 to e4c5bfa Compare April 1, 2022 19:56
@bartgol bartgol removed the AT: WIP label Apr 1, 2022
@bartgol
Copy link
Contributor Author

bartgol commented Apr 1, 2022 •
edited
Loading

There's still something to address, and I need some opinions here.

The current implementation allows our C++ code to write to the atm log file, but all Fortran routines still write to screen. In particular, Homme, is still printing all its output to screen, and it's quite a bit of output.

Unfortunately, I don't know if we can fix this, even if we managed to make C++ and Fortran use the same descriptor for the log file. Sharing descriptors between C and Fortran is not recommended, and I'm afraid we could mess up our logs. For instance, we may have C overwriting the last chunk written by F90, since it wasn't aware of another access to that same file. While I could modify the C++ logger, to make it "refresh" the file, to ensure it's not buffering the EOF position, it gets tricky with the Fortran side, where, even if I had a way to "refresh" the EOF position, I would have to modify tens/hundreds of calls to write.

I'm not sure how we should address this. A mixed C/Fortran program that needs to write to the same file from both C and Fortran is a bit of a niche problem, so you don't find much help online. I could try to play around a bit with a toy program, and see what happens, but my knowledge here is not very strong, and I might just "get lucky" in my test, and leave no "standard-backed" guarantee that it will work.

What do you guys think?

Tagging some v1/AD folks: @jgfouca @jeff-cohere @tcclevenger @PeterCaldwell @AaronDonahue @brhillman

@bartgol bartgol force-pushed the bartgol/logging-support branch from e4c5bfa to 8bef725 Compare April 1, 2022 20:05
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 1, 2022

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 1, 2022

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3027
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2918
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2268
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/logging-support
  • SHA: 8bef725
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 1, 2022

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 4 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3027
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2918
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2268
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Blake # 3027 (click to expand) 

[ 86%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_relaxation_timescale_unit_tests.cpp.o
[ 86%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_calc_liq_relaxation_timescale_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_nucleation_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_melting_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_evaporate_rain_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_cldliq_wet_growth_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_get_latent_heat_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_subgrid_variance_scaling_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_check_values_unit_tests.cpp.o
[ 87%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_incloud_mixingratios_unit_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_main_unit_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_supersat_conservation_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nr_conservation_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_nc_conservation_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ni_conservation_tests.cpp.o
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_ice_deposition_sublimation_tests.cpp.o
[ 88%] Linking CXX executable spa_main_test
[ 88%] Building CXX object src/physics/p3/tests/CMakeFiles/p3_tests.dir/p3_prevent_liq_supersaturation_tests.cpp.o
[ 88%] Built target spa_main_test
[ 88%] Linking CXX executable spa_one_to_one_remap_test
[ 88%] Built target spa_one_to_one_remap_test
[ 88%] Linking CXX executable spa_read_data_test
[ 88%] Built target spa_read_data_test
[ 88%] Linking CXX executable p3_tests
[ 88%] Built target p3_tests
[ 88%] Linking CXX executable io_test
[ 88%] Built target io_test
[ 88%] Linking CXX executable atm_proc
[ 88%] Built target atm_proc
[ 88%] Linking CXX executable field
[ 88%] Built target field
[ 88%] Linking CXX executable potential_temperature
[ 88%] Linking CXX static library libscream_control.a
[ 88%] Built target scream_control
[ 88%] Built target potential_temperature
[ 88%] Linking CXX executable output_restart_test
[ 88%] Built target output_restart_test
[ 88%] Linking CXX static library libscream_theta-l_kokkos_4_72_4.a
[ 88%] Built target scream_theta-l_kokkos_4_72_4
[ 88%] Linking CXX static library libscream_theta-l_kokkos_4_72_35.a
[ 88%] Built target scream_theta-l_kokkos_4_72_35
gmake: *** [all] Error 2

CMake Error at /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/ctest_script.cmake:79 (message):

Test had fails


===============================================================================

Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'


RUN: taskset -c 0-23 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_debug


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'debug_nopack_fpe'


RUN: taskset -c 48-71 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/debug_nopack_fpe


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'


RUN: taskset -c 24-47 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/full_sp_debug


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'


RUN: taskset -c 72-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/release/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream/ctest-build/release

Build type full_debug failed. Here's a list of failed tests:

Build type debug_nopack_fpe failed. Here's a list of failed tests:

Build type release failed. Here's a list of failed tests:

Error(s) occurred during test phase

OVERALL STATUS: FAIL

srun: error: blake01: task 0: Exited with exit code 1

salloc: Relinquishing job allocation 1021501

salloc: Job allocation 1021501 has been revoked.

Starting analysis on blake with cmd: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

RUN: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3027/scream/components/scream

blake failed


    

  • [[ 1 != 0 ]]
    • 

  • fails=0+1
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ blake == \m\a\p\p\y ]]
    • 

  • [[ 1 > 0 ]]
    • 

  • exit 1
    
Build step 'Execute shell' marked build as failure
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Blake] $ /bin/bash -le /tmp/jenkins8891793252278500825.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 2918 (click to expand) 

13/33 Test #13: p3_tests_omp1 ....................   Passed  256.83 sec
      Start 14: p3_tests_fail_omp1
14/33 Test #14: p3_tests_fail_omp1 ...............   Passed  304.43 sec
      Start 15: p3_run_and_cmp_cxx
15/33 Test #15: p3_run_and_cmp_cxx ...............   Passed    3.83 sec
      Start 16: p3_run_and_cmp_f90
16/33 Test #16: p3_run_and_cmp_f90 ...............   Passed    7.72 sec
      Start 17: p3_run_and_cmp_cxx_fail
17/33 Test #17: p3_run_and_cmp_cxx_fail ..........   Passed    6.12 sec
      Start 18: shoc_tests_omp1
18/33 Test #18: shoc_tests_omp1 ..................   Passed   29.12 sec
      Start 19: shoc_run_and_cmp_cxx
19/33 Test #19: shoc_run_and_cmp_cxx .............   Passed    0.82 sec
      Start 20: shoc_run_and_cmp_f90
20/33 Test #20: shoc_run_and_cmp_f90 .............   Passed   57.89 sec
      Start 21: spa_read_data_test_np1
21/33 Test #21: spa_read_data_test_np1 ...........   Passed    1.04 sec
      Start 22: spa_read_remap_test_np1
22/33 Test #22: spa_read_remap_test_np1 ..........   Passed    0.71 sec
      Start 23: spa_one_to_one_remap_test_np1
23/33 Test #23: spa_one_to_one_remap_test_np1 ....   Passed    0.91 sec
      Start 24: spa_main_test
24/33 Test #24: spa_main_test ....................   Passed    0.66 sec
      Start 25: potential_temperature
25/33 Test #25: potential_temperature ............   Passed    0.70 sec
      Start 26: ad_ut
26/33 Test #26: ad_ut ............................   Passed    0.74 sec
      Start 27: sc_ut
27/33 Test #27: sc_ut ............................   Passed    0.75 sec
      Start 28: fail
28/33 Test #28: fail .............................   Passed    0.62 sec
      Start 29: kokkos_fail
29/33 Test #29: kokkos_fail ......................   Passed    0.62 sec
      Start 30: p3_standalone_np1
30/33 Test #30: p3_standalone_np1 ................   Passed   35.05 sec
      Start 31: shoc_standalone_np1
31/33 Test #31: shoc_standalone_np1 ..............   Passed    0.93 sec
      Start 32: cld_fraction_standalone
32/33 Test #32: cld_fraction_standalone ..........   Passed    0.75 sec
      Start 33: spa_stand_alone_np1
33/33 Test #33: spa_stand_alone_np1 ..............   Passed   38.09 sec

100% tests passed, 0 tests failed out of 33


Label Time Summary:

cld_fraction    =   0.75 secproc (1 test)

diagnostics     =   0.70 secproc (1 test)

driver          =   1.49 secproc (2 tests)

fail            = 311.78 secproc (4 tests)

io              =   2.52 secproc (2 tests)

p3              = 613.98 secproc (6 tests)

physics         = 653.74 secproc (9 tests)

shoc            =   0.93 secproc (1 test)

spa             =  41.41 sec*proc (5 tests)


Total Test time (real) = 790.43 sec


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'


RUN: taskset -c 104-155 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/release


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'


RUN: taskset -c 0-51 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_debug


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'


RUN: taskset -c 52-103 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream/ctest-build/full_sp_debug

Build type full_debug failed. Here's a list of failed tests:

Build type release failed. Here's a list of failed tests:

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2918/scream/components/scream

weaver failed


    

  • [[ 1 != 0 ]]
    • 

  • fails=0+1
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • [[ 1 > 0 ]]
    • 

  • exit 1
    
Build step 'Execute shell' marked build as failure
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins6899517725110539263.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 2268 (click to expand) 

Total Test time (real) =  62.41 sec
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'debug_nopack_fpe'
===============================================================================
RUN: taskset -c 64-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/debug_nopack_fpe
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'
===============================================================================
RUN: taskset -c 0-31 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_debug
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'
===============================================================================
RUN: taskset -c 32-63 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/full_sp_debug
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'
===============================================================================
RUN: taskset -c 96-127 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream/ctest-build/release
Build type full_debug failed. Here's a list of failed tests:
Build type debug_nopack_fpe failed. Here's a list of failed tests:
Build type release failed. Here's a list of failed tests:
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy 
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy 
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2268/scream/components/scream
mappy failed
+ [[ 1 != 0 ]]
+ fails=0+1
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ mappy == \m\a\p\p\y ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
+ '[' -n 1520 ']'
+ ./scripts/git-merge-ref origin/master
+ [[ 0 != 0 ]]
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
+ ../../cime/scripts/create_test e3sm_scream_v1 --compiler=gnu9 -c -b master
Machine object has no machine defined
No project info available
create_test will do up to 2 tasks simultaneously
create_test will use up to 80 cores simultaneously
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220401_142505_bbq9ry
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220401_142505_bbq9ry
RUNNING TESTS:
  SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9
  SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.559324 seconds (PASS)
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.560111 seconds (PASS)
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.229166 seconds (PASS)
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.231491 seconds (PASS)
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.968165 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 3.348184 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 39.261012 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 39.045902 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 77.030903 seconds (PASS)
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 24 procs
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 78.333595 seconds (PASS)
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 24 procs
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 117.422483 seconds (PASS). [COMPLETED 1 of 2]
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 125.572038 seconds (PASS). [COMPLETED 2 of 2]
Waiting for tests to finish
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220401_142505_bbq9ry
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220401_142505_bbq9ry
test-scheduler took 248.74600481987 seconds
+ [[ 0 != 0 ]]
+ [[ 1 > 0 ]]
+ exit 1
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins5476063678016935600.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

@jeff-cohere
Copy link
Contributor

As far as I know, there's no standard way to share file descriptors between the two languages. There may be ways that work on certain platforms and/or compilers, but apart from that I think we're on our own. Does Fortran even offer access to the OS file descriptors it uses?

A standard approach would involve replacing Fortran write and print statements with calls to a function that writes output to a file in C++. The most obvious issue with this solution is that we wouldn't be able to use the formatting offered by the write and print statements. If that's no big deal, then that approach is pretty straightforward.

@bartgol
Copy link
Contributor Author

bartgol commented Apr 2, 2022

As far as I know, there's no standard way to share file descriptors between the two languages. There may be ways that work on certain platforms and/or compilers, but apart from that I think we're on our own. Does Fortran even offer access to the OS file descriptors it uses?

A standard approach would involve replacing Fortran write and print statements with calls to a function that writes output to a file in C++. The most obvious issue with this solution is that we wouldn't be able to use the formatting offered by the write and print statements. If that's no big deal, then that approach is pretty straightforward.

Yes, the ids of the descriptors are available. But replacing all calls to write with bridges to other functions is a mess, and not scalable. Besides, it requries to change Homme, on the premise that it is used inside a non-Fortran context. I would really like to avoid that path.

I fear we have to accept having all Homme output either suppressed, or written to screen (hence, to e3sm.log) rather than to atm.log.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 2, 2022

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3028
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2919
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2269
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/logging-support
  • SHA: 8bef725
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 2, 2022

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 4 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3028
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2919
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2269
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 8bef725
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Blake # 3028 (click to expand) 

[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_inplume_corr_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_computetemp_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_qs_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_s_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_buoyflux_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_liqflux_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_cloudvar_tests.cpp.o
[ 93%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pdf_compute_sgsliq_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_mom_ubycond_test.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_init_pot_test.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_unit_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_lbycond_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_moments_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_diag_second_shoc_moments_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_cldcheck_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_compute_shoc_vapor_tests.cpp.o
[ 94%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_update_prognostics_implicit_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_main_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_height_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_vd_shoc_decomp_and_solve_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_surf_temp_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_check_pblh_tests.cpp.o
[ 95%] Building CXX object src/physics/shoc/tests/CMakeFiles/shoc_tests.dir/shoc_pblintd_tests.cpp.o
[ 95%] Linking CXX executable shoc_tests
[ 95%] Built target shoc_tests
[ 95%] Linking CXX executable io_test
[ 95%] Built target io_test
[ 95%] Linking CXX executable atm_proc
[ 95%] Built target atm_proc
[ 95%] Linking CXX executable field
[ 95%] Built target field
[ 95%] Linking CXX executable output_restart_test
[ 95%] Linking CXX static library libscream_control.a
[ 95%] Built target scream_control
[ 95%] Built target output_restart_test
[ 95%] Linking CXX executable potential_temperature
[ 95%] Built target potential_temperature
[ 95%] Linking CXX static library libscream_theta-l_kokkos_4_72_4.a
[ 95%] Built target scream_theta-l_kokkos_4_72_4
[ 95%] Linking CXX static library libscream_theta-l_kokkos_4_72_35.a
[ 95%] Built target scream_theta-l_kokkos_4_72_35
gmake: *** [all] Error 2

CMake Error at /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/ctest_script.cmake:79 (message):

Test had fails


===============================================================================

Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'


RUN: taskset -c 0-23 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_debug


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'debug_nopack_fpe'


RUN: taskset -c 48-71 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/debug_nopack_fpe


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'


RUN: taskset -c 72-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/release/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/release


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'


RUN: taskset -c 24-47 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream/ctest-build/full_sp_debug

Build type full_debug failed. Here's a list of failed tests:

Build type debug_nopack_fpe failed. Here's a list of failed tests:

Build type release failed. Here's a list of failed tests:

Error(s) occurred during test phase

OVERALL STATUS: FAIL

srun: error: blake03: task 0: Exited with exit code 1

salloc: Relinquishing job allocation 1021540

salloc: Job allocation 1021540 has been revoked.

Starting analysis on blake with cmd: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

RUN: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3028/scream/components/scream

blake failed


    

  • [[ 1 != 0 ]]
    • 

  • fails=0+1
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ blake == \m\a\p\p\y ]]
    • 

  • [[ 1 > 0 ]]
    • 

  • exit 1
    
Build step 'Execute shell' marked build as failure
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Blake] $ /bin/bash -le /tmp/jenkins7228363417439500093.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 2919 (click to expand) 

13/33 Test #13: p3_tests_omp1 ....................   Passed  266.34 sec
      Start 14: p3_tests_fail_omp1
14/33 Test #14: p3_tests_fail_omp1 ...............   Passed  275.10 sec
      Start 15: p3_run_and_cmp_cxx
15/33 Test #15: p3_run_and_cmp_cxx ...............   Passed    3.81 sec
      Start 16: p3_run_and_cmp_f90
16/33 Test #16: p3_run_and_cmp_f90 ...............   Passed    7.71 sec
      Start 17: p3_run_and_cmp_cxx_fail
17/33 Test #17: p3_run_and_cmp_cxx_fail ..........   Passed    5.85 sec
      Start 18: shoc_tests_omp1
18/33 Test #18: shoc_tests_omp1 ..................   Passed   28.36 sec
      Start 19: shoc_run_and_cmp_cxx
19/33 Test #19: shoc_run_and_cmp_cxx .............   Passed    0.80 sec
      Start 20: shoc_run_and_cmp_f90
20/33 Test #20: shoc_run_and_cmp_f90 .............   Passed   57.95 sec
      Start 21: spa_read_data_test_np1
21/33 Test #21: spa_read_data_test_np1 ...........   Passed    1.09 sec
      Start 22: spa_read_remap_test_np1
22/33 Test #22: spa_read_remap_test_np1 ..........   Passed    0.82 sec
      Start 23: spa_one_to_one_remap_test_np1
23/33 Test #23: spa_one_to_one_remap_test_np1 ....   Passed    0.91 sec
      Start 24: spa_main_test
24/33 Test #24: spa_main_test ....................   Passed    0.69 sec
      Start 25: potential_temperature
25/33 Test #25: potential_temperature ............   Passed    0.70 sec
      Start 26: ad_ut
26/33 Test #26: ad_ut ............................   Passed    0.75 sec
      Start 27: sc_ut
27/33 Test #27: sc_ut ............................   Passed    0.73 sec
      Start 28: fail
28/33 Test #28: fail .............................   Passed    0.61 sec
      Start 29: kokkos_fail
29/33 Test #29: kokkos_fail ......................   Passed    0.61 sec
      Start 30: p3_standalone_np1
30/33 Test #30: p3_standalone_np1 ................   Passed   34.52 sec
      Start 31: shoc_standalone_np1
31/33 Test #31: shoc_standalone_np1 ..............   Passed    1.01 sec
      Start 32: cld_fraction_standalone
32/33 Test #32: cld_fraction_standalone ..........   Passed    0.75 sec
      Start 33: spa_stand_alone_np1
33/33 Test #33: spa_stand_alone_np1 ..............   Passed   38.30 sec

100% tests passed, 0 tests failed out of 33


Label Time Summary:

cld_fraction    =   0.75 secproc (1 test)

diagnostics     =   0.70 secproc (1 test)

driver          =   1.48 secproc (2 tests)

fail            = 282.17 secproc (4 tests)

io              =   2.64 secproc (2 tests)

p3              = 593.33 secproc (6 tests)

physics         = 633.39 secproc (9 tests)

shoc            =   1.01 secproc (1 test)

spa             =  41.80 sec*proc (5 tests)


Total Test time (real) = 769.72 sec


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'


RUN: taskset -c 52-103 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_sp_debug


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'


RUN: taskset -c 104-155 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/release


Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'


RUN: taskset -c 0-51 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream/ctest-build/full_debug

Build type full_debug failed. Here's a list of failed tests:

Build type release failed. Here's a list of failed tests:

Error(s) occurred during test phase

OVERALL STATUS: FAIL

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2919/scream/components/scream

weaver failed


    

  • [[ 1 != 0 ]]
    • 

  • fails=0+1
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • [[ 1 > 0 ]]
    • 

  • exit 1
    
Build step 'Execute shell' marked build as failure
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins1737343844976459035.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 2269 (click to expand) 

Total Test time (real) =  63.43 sec
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_debug'
===============================================================================
RUN: taskset -c 0-31 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_debug
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'debug_nopack_fpe'
===============================================================================
RUN: taskset -c 64-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/debug_nopack_fpe
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'full_sp_debug'
===============================================================================
RUN: taskset -c 32-63 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/full_sp_debug
===============================================================================
Testing '8bef725251e9558e0d9fd7aef3a57f33560ada9a' for test 'release'
===============================================================================
RUN: taskset -c 96-127 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release/data" '
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream/ctest-build/release
Build type full_debug failed. Here's a list of failed tests:
Build type debug_nopack_fpe failed. Here's a list of failed tests:
Build type release failed. Here's a list of failed tests:
Error(s) occurred during test phase
OVERALL STATUS: FAIL
Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy 
RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy 
FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2269/scream/components/scream
mappy failed
+ [[ 1 != 0 ]]
+ fails=0+1
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ mappy == \m\a\p\p\y ]]
+ [[ 0 == 1 ]]
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
+ '[' -n 1520 ']'
+ ./scripts/git-merge-ref origin/master
+ [[ 0 != 0 ]]
+ [[ 0 == 1 ]]
+ [[ 1 == 1 ]]
+ ../../cime/scripts/create_test e3sm_scream_v1 --compiler=gnu9 -c -b master
Machine object has no machine defined
No project info available
create_test will do up to 2 tasks simultaneously
create_test will use up to 80 cores simultaneously
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220401_201714_5wz45i
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220401_201714_5wz45i
RUNNING TESTS:
  SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9
  SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.558332 seconds (PASS)
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.564593 seconds (PASS)
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.224002 seconds (PASS)
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.226465 seconds (PASS)
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 3.223017 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 6.828417 seconds (PASS)
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 39.518253 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 16 procs
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 38.641279 seconds (PASS)
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 77.255089 seconds (PASS)
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 24 procs
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 76.486492 seconds (PASS)
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 24 procs
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 146.851420 seconds (PASS). [COMPLETED 1 of 2]
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 148.694720 seconds (PASS). [COMPLETED 2 of 2]
Waiting for tests to finish
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220401_201714_5wz45i
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 RUN
    Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220401_201714_5wz45i
test-scheduler took 271.062513589859 seconds
+ [[ 0 != 0 ]]
+ [[ 1 > 0 ]]
+ exit 1
Build step 'Execute shell' marked build as failure
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins6759433600195279389.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 2, 2022

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@PeterCaldwell
Copy link
Contributor

Oh man, that sucks. The fact that we're trying to write logs from 2 different languages didn't cross my mind.

I think it's fine for F90 to write to e3sm.log and c++ to write to atm.log for the v1 release - we're trying to create something that works now, not something that is perfect. This isn't an acceptable long-term solution though.

One question is: why doesn't HOMME write to e3sm.log in EAMf90 runs? At least I'm assuming it doesn't... It may be acceptable as a long-term solution to have both an atm.cxx.log and an atm.f90.log file...

I'm also unclear why replacing all printf and write statements with wrappers wouldn't be scalable (though I agree it would be really annoying). Maybe we could search/replace all write and printf statements with my_write and my_printf which just call the intrinsic write and print for fortran builds and call C++ versions that take the same syntax for C++ builds?

@jeff-cohere
Copy link
Contributor

I'm also unclear why replacing all printf and write statements with wrappers wouldn't be scalable (though I agree it would be really annoying). Maybe we could search/replace all write and printf statements with my_write and my_printf which just call the intrinsic write and print for fortran builds and call C++ versions that take the same syntax for C++ builds?

Your search/replace idea is practical for languages for which print/write/etc are functions and not core-language commands. But it's not easy when they are commands (as they are in Fortran), because the commands have their own special syntax, which is difficult to shoehorn into a function call. This was a big driver for the most controversial change early on in the switch from Python 2 to Python 3 (which changed print from a command to a function, which upset basically all the science folks).

@bartgol
Copy link
Contributor Author

bartgol commented Apr 4, 2022

One question is: why doesn't HOMME write to e3sm.log in EAMf90 runs? At least I'm assuming it doesn't... It may be acceptable as a long-term solution to have both an atm.cxx.log and an atm.f90.log file...

Homme writes to atm.log, since it has access to the F90 file descriptor. As I said, we could try to do the same when Homme runs through EAMxx (I would need a micro PR in E3SM, but that's doable). The issue is having 2 output streams open to the same file, one in F90 and one in CXX. While we do have some control on the CXX stream, I don't think we can guarantee anything regarding the F90 buffer status when we write. I should probably dig a bit on this, although as I mentioned, the online documentation for "a mixed C/F90 program, with a persistent buffer in both language open for write" is scarse (at best). And working on a toy problem might be misleading, since I suspect some sync issues are obvious only when running real jobs, with messages of variable size, which can fill up the F90 buffer quickly but the CXX one slowly (or viceversa).

Of course, there's also the alternative of assigning Homme to a blackhole stream. Or even a separate log file (like atm_dyn.log or something).

@jeff-cohere
Copy link
Contributor

Shared I/O between C++ and Fortran scares me a bit. But at least Fortran 2003 does seem to let you explicitly flush your writes.

@PeterCaldwell
Copy link
Contributor

Yeah, I'm kind of creeped out by shared C++/Fortran writes too. I'd suggest just letting dyn write its own log file and being done with it for the moment.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 5, 2022

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 4 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3037
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1ac2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2927
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1ac2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2275
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA 1ac2417
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 4708d3c
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Blake # 3037 (click to expand) 

 > /home/projects/x86-64/git/2.9.4/bin/git config --get submodule.externals/mct.url # timeout=10
 > /home/projects/x86-64/git/2.9.4/bin/git remote # timeout=10
 > /home/projects/x86-64/git/2.9.4/bin/git config --get remote.origin.url # timeout=10
 > /home/projects/x86-64/git/2.9.4/bin/git config -f .gitmodules --get submodule.externals/mct.path # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive components/elm/src/external_models/fates # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive components/elm/src/external_models/mpp # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive components/elm/src/external_models/sbetr # timeout=10
ERROR: Timeout after 10 minutes
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive externals/kokkos # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive components/eam/src/physics/rrtmgp/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive externals/scorpio # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive externals/scorpio_classic # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive components/eam/src/physics/cosp2/external # timeout=10
using GIT_SSH to set credentials jgfouca github key
 > /home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive cime # timeout=10
ERROR: Timeout after 10 minutes
hudson.plugins.git.GitException: Command "/home/projects/x86-64/git/2.9.4/bin/git submodule update --init --recursive cime" returned status code 143:
stdout: 
stderr: Cloning into '/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3037/scream/cime'...

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2671)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2096)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$500(CliGitAPIImpl.java:84)
at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.lambda$execute$0(CliGitAPIImpl.java:1505)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
at java.util.concurrent.ExecutorCompletionService.submit(ExecutorCompletionService.java:181)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.submitRemainingCommand(GitCommandsExecutor.java:98)
at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:92)



Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to blake

at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)

at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)

at hudson.remoting.Channel.call(Channel.java:1001)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)

at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)

at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.base/java.lang.reflect.Method.invoke(Method.java:566)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)

at com.sun.proxy.$Proxy65.execute(Unknown Source)

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:160)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1387)

at hudson.scm.SCM.checkout(SCM.java:505)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1213)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:637)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:509)

at hudson.model.Run.execute(Run.java:1888)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)

at hudson.model.ResourceController.execute(ResourceController.java:99)

at hudson.model.Executor.run(Executor.java:432)

Caused: hudson.plugins.git.GitException

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.checkResult(GitCommandsExecutor.java:110)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:91)

at org.jenkinsci.plugins.gitclient.cgit.GitCommandsExecutor.invokeAll(GitCommandsExecutor.java:49)

at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$7.execute(CliGitAPIImpl.java:1508)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)

at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)

at hudson.remoting.UserRequest.perform(UserRequest.java:211)

at hudson.remoting.UserRequest.perform(UserRequest.java:54)

at hudson.remoting.Request$2.run(Request.java:376)

at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

Caused: java.io.IOException: Could not perform submodule update

at hudson.plugins.git.extensions.impl.SubmoduleOption.onCheckoutCompleted(SubmoduleOption.java:165)

at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1387)

at hudson.scm.SCM.checkout(SCM.java:505)

at hudson.model.AbstractProject.checkout(AbstractProject.java:1213)

at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:637)

at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)

at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:509)

at hudson.model.Run.execute(Run.java:1888)

at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)

at hudson.model.ResourceController.execute(ResourceController.java:99)

at hudson.model.Executor.run(Executor.java:432)

Performing Post build task...

Match found for : : True

Logical operation result is TRUE

Running script  : #!/bin/bash -le


cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Blake] $ /bin/bash -le /tmp/jenkins3884529214203627578.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 2927 (click to expand) 

26/48 Test #26: spa_read_remap_test_np1 ............   Passed    0.72 sec
      Start 27: spa_one_to_one_remap_test_np1
27/48 Test #27: spa_one_to_one_remap_test_np1 ......   Passed    0.94 sec
      Start 28: spa_main_test
28/48 Test #28: spa_main_test ......................   Passed    0.69 sec
      Start 29: potential_temperature
29/48 Test #29: potential_temperature ..............   Passed    0.69 sec
      Start 30: ad_ut
30/48 Test #30: ad_ut ..............................   Passed    0.79 sec
      Start 31: sc_ut
31/48 Test #31: sc_ut ..............................   Passed    0.86 sec
      Start 32: fail
32/48 Test #32: fail ...............................   Passed    0.65 sec
      Start 33: kokkos_fail
33/48 Test #33: kokkos_fail ........................   Passed    0.65 sec
      Start 34: homme_standalone_np1
34/48 Test #34: homme_standalone_np1 ...............   Passed    5.10 sec
      Start 35: p3_standalone_np1
35/48 Test #35: p3_standalone_np1 ..................   Passed   25.82 sec
      Start 36: shoc_standalone_np1
36/48 Test #36: shoc_standalone_np1 ................   Passed    1.06 sec
      Start 37: cld_fraction_standalone
37/48 Test #37: cld_fraction_standalone ............   Passed    0.82 sec
      Start 38: spa_stand_alone_np1
38/48 Test #38: spa_stand_alone_np1 ................   Passed   37.72 sec
      Start 39: rrtmgp_standalone_unit
39/48 Test #39: rrtmgp_standalone_unit .............   Passed    1.02 sec
      Start 40: rrtmgp_standalone_np1
40/48 Test #40: rrtmgp_standalone_np1 ..............   Passed    1.30 sec
      Start 41: shoc_cld_p3_rrtmgp_np1
41/48 Test #41: shoc_cld_p3_rrtmgp_np1 .............   Passed   27.49 sec
      Start 42: shoc_cld_spa_p3_rrtmgp_np1
42/48 Test #42: shoc_cld_spa_p3_rrtmgp_np1 .........   Passed   46.83 sec
      Start 43: homme_shoc_cld_p3_rrtmgp_np1
43/48 Test #43: homme_shoc_cld_p3_rrtmgp_np1 .......   Passed   36.99 sec
      Start 44: model_baseline
44/48 Test #44: model_baseline .....................   Passed   35.04 sec
      Start 45: model_initial
45/48 Test #45: model_initial ......................   Passed    7.01 sec
      Start 46: model_restart
46/48 Test #46: model_restart ......................   Passed    8.22 sec
      Start 47: model_restart_check
47/48 Test #47: model_restart_check ................   Passed    0.10 sec
      Start 48: homme_shoc_cld_spa_p3_rrtmgp_np1
48/48 Test #48: homme_shoc_cld_spa_p3_rrtmgp_np1 ...   Passed   56.43 sec

100% tests passed, 0 tests failed out of 48


Label Time Summary:

cld             = 167.74 secproc (4 tests)

cld_fraction    =   0.82 secproc (1 test)

diagnostics     =   0.69 secproc (1 test)

driver          = 100.17 secproc (5 tests)

dynamics        = 102.68 secproc (5 tests)

fail            = 284.32 secproc (4 tests)

io              =  12.35 secproc (3 tests)

p3              = 899.66 secproc (10 tests)

physics         = 943.73 secproc (17 tests)

rrtmgp          = 172.20 secproc (8 tests)

shoc            = 168.80 secproc (5 tests)

spa             =  88.10 secproc (6 tests)


Total Test time (real) = 1178.32 sec


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'release'


RUN: taskset -c 104-155 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/release/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/release


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'full_debug'


RUN: taskset -c 0-51 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_debug


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'full_sp_debug'


RUN: taskset -c 52-103 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=52 CTEST_PARALLEL_LEVEL=1 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/cmake/machine-files/weaver.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105 -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105 -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicxx -DCMAKE_C_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpicc -DCMAKE_Fortran_COMPILER=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=1 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/weaver/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream/ctest-build/full_sp_debug

OVERALL STATUS: PASS

Starting analysis on weaver with cmd: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

RUN: cd /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream && module purge && module load cuda/10.1.105 ucx/1.6.0 git/2.10.1 python/3.7.3 cmake/3.18.0 && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/mpfr/install/weaver/lib:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib64:${LD_LIBRARY_PATH} && export LD_LIBRARY_PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/lib:${LD_LIBRARY_PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/gcc/install/weaver/gcc/8.5.0/libexec/gcc/powerpc64le-unknown-linux-gnu/8.5.0:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/openmpi/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/weaver/gcc/8.5.0/cuda/10.1.105/bin:${PATH} && export PATH=/ascldap/users/projects/e3sm/scream/libs/wget/bin:$PATH && true && bsub -I -q rhel7W -n 4 ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m weaver

FROM: /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2927/scream/components/scream

Completed analysis on weaver


    

  • [[ 0 != 0 ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ weaver == \m\a\p\p\y ]]
    • 

  • [[ 0 > 0 ]]
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins4088501756387630148.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 2275 (click to expand) 

fail            = 528.75 sec*proc (19 tests)
io              = 128.22 sec*proc (12 tests)
p3              = 5934.91 sec*proc (67 tests)
physics         = 6858.77 sec*proc (92 tests)
rrtmgp          = 4931.76 sec*proc (38 tests)
shoc            = 4486.12 sec*proc (35 tests)
spa             = 1221.61 sec*proc (27 tests)

Total Test time (real) = 335.17 sec


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'full_sp_debug'


RUN: taskset -c 32-63 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_sp_debug


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'debug_nopack_fpe'


RUN: taskset -c 64-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/debug_nopack_fpe


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'full_debug'


RUN: taskset -c 0-31 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/full_debug


Testing '1ac24171a1c3307cf409c06369ee9f27c8e163b3' for test 'release'


RUN: taskset -c 96-127 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream/ctest-build/release

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2275/scream/components/scream

Completed analysis on mappy


    

  • [[ 0 != 0 ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ mappy == \m\a\p\p\y ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 1 == 1 ]]
    • 

  • '[' -n 1520 ']'
    • 

  • ./scripts/git-merge-ref origin/master
    • 

  • [[ 0 != 0 ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 1 == 1 ]]
    • 

  • ../../cime/scripts/create_test e3sm_scream_v1 --compiler=gnu9 -c -b master
    
Machine object has no machine defined
    
No project info available
    
create_test will do up to 2 tasks simultaneously
    
create_test will use up to 80 cores simultaneously
    
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220405_131931_a3di80
    
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220405_131931_a3di80
    
RUNNING TESTS:
    
SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9
    
SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9
    
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.562621 seconds (PASS)
    
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.565601 seconds (PASS)
    
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.224739 seconds (PASS)
    
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.227284 seconds (PASS)
    
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.610122 seconds (PASS)
    
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 2.988030 seconds (PASS)
    
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 39.399397 seconds (PASS)
    
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 39.613895 seconds (PASS)
    
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
    
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 16 procs
    
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 81.166273 seconds (PASS)
    
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 24 procs
    
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 84.614760 seconds (PASS)
    
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 24 procs
    
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 120.527242 seconds (PASS). [COMPLETED 1 of 2]
    
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 127.556660 seconds (PASS). [COMPLETED 2 of 2]
    
Waiting for tests to finish
    
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220405_131931_a3di80
    
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 RUN
    
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220405_131931_a3di80
    
test-scheduler took 256.75596618652344 seconds
    • 

  • [[ 0 != 0 ]]
    • 

  • [[ 0 > 0 ]]
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins4009358409828041924.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@bartgol
Copy link
Contributor Author

bartgol commented Apr 5, 2022

Blake is currently experiencing some issue, with commands taking a long time to complete on the login node, causing timeout during the git clone phase of our jenkins jobs. I manually tested this PR on a compute node, and all tests pass. Mappy and weaver are already passing.

@PeterCaldwell @AaronDonahue If you are ok with the current status of the PR, feel free to manually merge.

@PeterCaldwell
Copy link
Contributor

I'm happy with the PR from the discussions we've had, but don't have bandwidth to read through it in depth before merging it. If you want me to merge it without reading, I'm happy to do that. Otherwise I will let it sit until Jim or Aaron gets to it...

@bartgol bartgol requested a review from jgfouca April 5, 2022 22:20
@bartgol
Copy link
Contributor Author

bartgol commented Apr 5, 2022

No, merging without review is not appropriate, I think. I added JIm as reviewer. As soon as one between him and Aaron can take a look, we can move forward.

@bartgol bartgol removed the request for review from PeterCaldwell April 5, 2022 22:21
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 6, 2022

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Weaver to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 6, 2022

Status Flag 'Pull Request AutoTester' - Failure: Timed out waiting for job SCREAM_PullRequest_Autotester_Mappy to start: Total Wait = 1803

  • Other jobs have been previously started - We must stop them...

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 6, 2022

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 6, 2022

Status Flag 'Pull Request AutoTester' - GitHub reports Mergeable status = False

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 7, 2022

Status Flag 'Pre-Test Inspection' - Auto Inspected - Inspection Is Not Necessary for this Pull Request.

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 7, 2022

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3061
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2944
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2290
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/logging-support
  • SHA: e7d0e13
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 7, 2022

Status Flag 'Pull Request AutoTester' - Jenkins Testing: 1 or more Jobs FAILED

Note: Testing will normally be attempted again in approx. 4 Hrs. If a change to the PR source branch occurs, the testing will be attempted again on next available autotester run.

Pull Request Auto Testing has FAILED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3061
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2944
  • Status: FAILED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2290
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Blake # 3061 (click to expand) 

        Start 100: shoc_tests_omp16
189/191 Test #100: shoc_tests_omp16 ..............................   Passed    0.57 sec
        Start 101: shoc_run_and_cmp_cxx_omp16
190/191 Test #101: shoc_run_and_cmp_cxx_omp16 ....................   Passed    0.31 sec
        Start 102: shoc_run_and_cmp_f90_omp16
191/191 Test #102: shoc_run_and_cmp_f90_omp16 ....................   Passed    0.33 sec

100% tests passed, 0 tests failed out of 191


Label Time Summary:

cld             = 555.20 secproc (28 tests)

cld_fraction    =   0.64 secproc (1 test)

diagnostics     =   0.57 secproc (1 test)

driver          = 307.15 secproc (23 tests)

dynamics        = 323.08 secproc (29 tests)

fail            = 202.45 secproc (18 tests)

io              =  92.98 secproc (12 tests)

p3              = 1173.10 secproc (71 tests)

physics         = 1353.16 secproc (95 tests)

rrtmgp          = 660.06 secproc (38 tests)

shoc            = 577.63 secproc (36 tests)

spa             = 272.30 secproc (27 tests)


Total Test time (real) =  82.50 sec

181/188 Test #171: homme_shoc_cld_p3_rrtmgp_np1 ..................   Passed  457.53 sec

Start 175: homme_shoc_cld_p3_rrtmgp_np1_vs_np2_bfb

182/188 Test #175: homme_shoc_cld_p3_rrtmgp_np1_vs_np2_bfb .......   Passed    0.11 sec

Start 176: homme_shoc_cld_p3_rrtmgp_np1_vs_np3_bfb

183/188 Test #176: homme_shoc_cld_p3_rrtmgp_np1_vs_np3_bfb .......   Passed    0.08 sec

Start 177: homme_shoc_cld_p3_rrtmgp_np1_vs_np4_bfb

184/188 Test #177: homme_shoc_cld_p3_rrtmgp_np1_vs_np4_bfb .......   Passed    0.09 sec

185/188 Test #182: homme_shoc_cld_spa_p3_rrtmgp_np1 ..............   Passed  486.55 sec

Start 186: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np2_bfb

186/188 Test #186: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np2_bfb ...   Passed    0.17 sec

Start 187: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np3_bfb

187/188 Test #187: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np3_bfb ...   Passed    0.16 sec

Start 188: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np4_bfb

188/188 Test #188: homme_shoc_cld_spa_p3_rrtmgp_np1_vs_np4_bfb ...   Passed    0.16 sec


100% tests passed, 0 tests failed out of 188


Label Time Summary:

cld             = 6408.97 secproc (28 tests)

cld_fraction    =   1.56 secproc (1 test)

diagnostics     =   1.29 secproc (1 test)

driver          = 5075.48 secproc (23 tests)

dynamics        = 5105.99 secproc (29 tests)

fail            = 627.44 secproc (19 tests)

io              = 123.52 secproc (12 tests)

p3              = 8160.06 secproc (68 tests)

physics         = 9388.22 secproc (92 tests)

rrtmgp          = 7190.00 secproc (38 tests)

shoc            = 6476.72 secproc (36 tests)

spa             = 1622.37 secproc (27 tests)


Total Test time (real) = 620.61 sec


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'full_sp_debug'


RUN: taskset -c 24-47 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_sp_debug


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'full_debug'


RUN: taskset -c 0-23 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/full_debug/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/full_debug


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'debug_nopack_fpe'


RUN: taskset -c 48-71 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/debug_nopack_fpe


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'release'


RUN: taskset -c 72-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=24 CTEST_PARALLEL_LEVEL=24 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/cmake/machine-files/blake.cmake -DNetCDF_Fortran_PATH=/home/projects/e3sm/scream/libs/netcdf-fortran/install/blake -DNetCDF_C_PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicxx -DCMAKE_C_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpicc -DCMAKE_Fortran_COMPILER=/home/projects/x86-64-skylake/openmpi/2.1.2/intel/19.3.199/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=24 -DSCREAM_TEST_DATA_DIR=/home/projects/e3sm/scream/pr-autotester/master-baselines/blake/release/data" '

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream/ctest-build/release

OVERALL STATUS: PASS

salloc: Relinquishing job allocation 1021855

salloc: Job allocation 1021855 has been revoked.

Starting analysis on blake with cmd: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

RUN: cd /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream && module purge && module load python/3.8.8/gcc/10.2.0 && module unload gcc/10.2.0 && module load openmpi/2.1.2 zlib git/2.9.4 cmake/3.19.3 && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-fortran/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/netcdf-c/install/blake/bin:$PATH && export PATH=/ascldap/users/projects/e3sm/scream/libs/pnetcdf/install/blake/bin:$PATH && export INTEL_LICENSE_FILE=/home/projects/x86-64/intel/licenses/USE_SERVER.lic && export OMP_PROC_BIND=spread && true && salloc -N 1 srun -n1 --preserve-env ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m blake

FROM: /home/e3sm-jenkins/blake/workspace/SCREAM_PullRequest_Autotester_Blake/3061/scream/components/scream

Completed analysis on blake


    

  • [[ 0 != 0 ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ blake == \m\a\p\p\y ]]
    • 

  • [[ 0 > 0 ]]
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Blake] $ /bin/bash -le /tmp/jenkins6680092316204275807.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Weaver # 2944 (click to expand) 

Running as SYSTEM
Building remotely on weaver in workspace /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver
The recommended git tool is: NONE
using credential cf1815d3-62c3-4157-a38d-0ad235093618
Wiping out workspace first.
Cloning the remote Git repository
Honoring refspec on initial clone
Cloning repository https://github.com/E3SM-Project/scream.git
 > /home/projects/ppc64le/git/2.10.1/bin/git init /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2944/scream # timeout=10
ERROR: Error cloning remote repo 'origin'
hudson.plugins.git.GitException: Could not init /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2944/scream
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1042)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:797)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:158)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:151)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:376)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
	Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to weaver
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1797)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1001)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:143)
		at jdk.internal.reflect.GeneratedMethodAccessor37.invoke(Unknown Source)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:129)
		at com.sun.proxy.$Proxy67.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1224)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1302)
		at hudson.scm.SCM.checkout(SCM.java:505)
		at hudson.model.AbstractProject.checkout(AbstractProject.java:1213)
		at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:637)
		at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:85)
		at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:509)
		at hudson.model.Run.execute(Run.java:1888)
		at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
		at hudson.model.ResourceController.execute(ResourceController.java:99)
		at hudson.model.Executor.run(Executor.java:432)
Caused by: hudson.plugins.git.GitException: Error performing git command: /home/projects/ppc64le/git/2.10.1/bin/git init /home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2944/scream
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2679)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2601)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2597)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommand(CliGitAPIImpl.java:1968)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$5.execute(CliGitAPIImpl.java:1040)
	... 11 more
Caused by: java.io.IOException: Cannot run program "/home/projects/ppc64le/git/2.10.1/bin/git" (in directory "/home/e3sm-jenkins/weaver/workspace/SCREAM_PullRequest_Autotester_Weaver/2944/scream"): error=2, No such file or directory
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1128)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1071)
	at hudson.Proc$LocalProc.(Proc.java:252)
	at hudson.Proc$LocalProc.(Proc.java:221)
	at hudson.Launcher$LocalLauncher.launch(Launcher.java:995)
	at hudson.Launcher$ProcStarter.start(Launcher.java:507)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2664)
	... 15 more
Caused by: java.io.IOException: error=2, No such file or directory
	at java.base/java.lang.ProcessImpl.forkAndExec(Native Method)
	at java.base/java.lang.ProcessImpl.(ProcessImpl.java:340)
	at java.base/java.lang.ProcessImpl.start(ProcessImpl.java:271)
	at java.base/java.lang.ProcessBuilder.start(ProcessBuilder.java:1107)
	... 21 more
ERROR: Error cloning remote repo 'origin'
Performing Post build task...
Match found for : : True
Logical operation result is TRUE
Running script  : #!/bin/bash -le

cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Weaver] $ /bin/bash -le /tmp/jenkins4888623656318212967.sh

/tmp/jenkins4888623656318212967.sh: line 5: ./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh: No such file or directory

POST BUILD TASK : FAILURE

END OF POST BUILD TASK : 0

Sending e-mails to: lbertag@sandia.gov

Finished: FAILURE
Console Output (last 100 lines) : SCREAM_PullRequest_Autotester_Mappy # 2290 (click to expand) 

fail            = 635.67 sec*proc (19 tests)
io              = 261.31 sec*proc (12 tests)
p3              = 6069.53 sec*proc (68 tests)
physics         = 6990.32 sec*proc (92 tests)
rrtmgp          = 4963.79 sec*proc (38 tests)
shoc            = 4506.11 sec*proc (36 tests)
spa             = 1231.58 sec*proc (27 tests)

Total Test time (real) = 435.46 sec


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'full_sp_debug'


RUN: taskset -c 32-63 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_sp_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_sp_debug -DBUILD_NAME_MOD=full_sp_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_DOUBLE_PRECISION=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_sp_debug/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_sp_debug


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'release'


RUN: taskset -c 96-127 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/release/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/release -DBUILD_NAME_MOD=release -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Release -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/release/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/release


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'debug_nopack_fpe'


RUN: taskset -c 64-95 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/debug_nopack_fpe/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/debug_nopack_fpe -DBUILD_NAME_MOD=debug_nopack_fpe -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DSCREAM_PACK_SIZE=1 -DSCREAM_SMALL_PACK_SIZE=1 -DSCREAM_ENABLE_BASELINE_TESTS=False -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/debug_nopack_fpe


Testing 'e7d0e13e3930ba110bbbb1ffc9e3e26edd3e4b82' for test 'full_debug'


RUN: taskset -c 0-31 sh -c 'SCREAM_BUILD_PARALLEL_LEVEL=32 CTEST_PARALLEL_LEVEL=32 ctest -V --output-on-failure --resource-spec-file /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_debug/ctest_resource_file.json -DNO_SUBMIT=True -DBUILD_WORK_DIR=/home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_debug -DBUILD_NAME_MOD=full_debug -S /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/ctest_script.cmake -DCMAKE_COMMAND="-C /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/cmake/machine-files/mappy.cmake -DNetCDF_Fortran_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DNetCDF_C_PATH=/projects/sems/install/rhel7-x86_64/sems/tpl/netcdf/4.7.3/gcc/9.2.0/openmpi/4.0.2/parallel -DCMAKE_BUILD_TYPE=Debug -DEKAT_DEFAULT_BFB=True -DEKAT_DISABLE_TPL_WARNINGS='ON' -DSCREAM_AUTOTESTER='ON' -DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicxx -DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpicc -DCMAKE_Fortran_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/9.2.0/openmpi/4.0.2/bin/mpifort -DSCREAM_DYNAMICS_DYCORE=HOMME -DSCREAM_TEST_MAX_TOTAL_THREADS=32 -DSCREAM_TEST_DATA_DIR=/sems-data-store/ACME/baselines/scream/master-baselines/full_debug/data" '

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream/ctest-build/full_debug

OVERALL STATUS: PASS

Starting analysis on mappy with cmd: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy

RUN: cd /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream && module purge && module load sems-archive-env sems-archive-gcc/9.2.0 sems-archive-cmake/3.19.1 sems-archive-git/2.10.1 sems-archive-openmpi/4.0.2 sems-archive-netcdf/4.7.3/parallel && export OMP_PROC_BIND=spread && true &&  ./scripts/test-all-scream --baseline-dir AUTO $compiler -c EKAT_DISABLE_TPL_WARNINGS=ON -c SCREAM_AUTOTESTER=ON -p -i -m mappy

FROM: /home/e3sm-jenkins/jenkins-ws/workspace/SCREAM_PullRequest_Autotester_Mappy/2290/scream/components/scream

Completed analysis on mappy


    

  • [[ 0 != 0 ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ -n '' ]]
    • 

  • [[ mappy == \m\a\p\p\y ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 1 == 1 ]]
    • 

  • '[' -n 1520 ']'
    • 

  • ./scripts/git-merge-ref origin/master
    • 

  • [[ 0 != 0 ]]
    • 

  • [[ 0 == 1 ]]
    • 

  • [[ 1 == 1 ]]
    • 

  • ../../cime/scripts/create_test e3sm_scream_v1 --compiler=gnu9 -c -b master
    
Machine object has no machine defined
    
No project info available
    
create_test will do up to 2 tasks simultaneously
    
create_test will use up to 80 cores simultaneously
    
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220407_124711_3hw847
    
Creating test directory /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220407_124711_3hw847
    
RUNNING TESTS:
    
SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9
    
SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9
    
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Starting CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.546476 seconds (PASS)
    
Finished CREATE_NEWCASE for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.552897 seconds (PASS)
    
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Starting XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 0.222347 seconds (PASS)
    
Finished XML for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 0.224006 seconds (PASS)
    
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Starting SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 2.670415 seconds (PASS)
    
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 1 procs
    
Finished SETUP for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 3.021426 seconds (PASS)
    
Starting SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 1 procs
    
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 40.206608 seconds (PASS)
    
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 16 procs
    
Finished SHAREDLIB_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 40.394885 seconds (PASS)
    
Starting MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 16 procs
    
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 84.305890 seconds (PASS)
    
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 with 24 procs
    
Finished MODEL_BUILD for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 87.407992 seconds (PASS)
    
Starting RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 with 24 procs
    
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 in 127.148295 seconds (PASS). [COMPLETED 1 of 2]
    
Finished RUN for test SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 in 135.418118 seconds (PASS). [COMPLETED 2 of 2]
    
Waiting for tests to finish
    
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9 RUN
    
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000-SCREAMv1-AQP1.mappy_gnu9.C.20220407_124711_3hw847
    
PASS SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9 RUN
    
Case dir: /ascldap/users/e3sm-jenkins/acme/scratch/SMS_D_Ln2_P24x1.ne4_ne4.F2000SCREAMv1.mappy_gnu9.C.20220407_124711_3hw847
    
test-scheduler took 268.53588032722473 seconds
    • 

  • [[ 0 != 0 ]]
    • 

  • [[ 0 > 0 ]]
    
Performing Post build task...
    
Match found for : : True
    
Logical operation result is TRUE
    
Running script  : #!/bin/bash -le
    • 



cd $WORKSPACE/${BUILD_ID}/


./scream/components/scream/scripts/jenkins/jenkins_cleanup.sh

[SCREAM_PullRequest_Autotester_Mappy] $ /bin/bash -le /tmp/jenkins2528991311675644892.sh

POST BUILD TASK : SUCCESS

END OF POST BUILD TASK : 0

Finished: SUCCESS

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 7, 2022

Status Flag 'Pull Request AutoTester' - Testing Jenkins Projects:

Pull Request Auto Testing STARTING (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3064
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2947
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2293
  • Status: STARTED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Using Repos:

Repo: SCREAM (E3SM-Project/scream)
  • Branch: bartgol/logging-support
  • SHA: e7d0e13
  • Mode: TEST_REPO

Pull Request Author: bartgol

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

Status Flag 'Pull Request AutoTester' - Jenkins Testing: all Jobs PASSED

Pull Request Auto Testing has PASSED (click to expand)

Build Information

Test Name: SCREAM_PullRequest_Autotester_Blake

  • Build Num: 3064
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Weaver

  • Build Num: 2947
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

Build Information

Test Name: SCREAM_PullRequest_Autotester_Mappy

  • Build Num: 2293
  • Status: PASSED

Jenkins Parameters

Parameter Name Value
PR_LABELS enhancement;SCREAMv1
PULLREQUESTNUM 1520
SCREAM_SOURCE_BRANCH bartgol/logging-support
SCREAM_SOURCE_REPO https://github.com/E3SM-Project/scream
SCREAM_SOURCE_SHA e7d0e13
SCREAM_TARGET_BRANCH master
SCREAM_TARGET_REPO https://github.com/E3SM-Project/scream
SCREAM_TARGET_SHA 5de80e0
TEST_REPO_ALIAS SCREAM

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

Status Flag 'Pre-Merge Inspection' - - This Pull Request Requires Inspection... The code must be inspected by a member of the Team before Testing/Merging
THE LAST COMMIT TO THIS PULL REQUEST HAS NOT BEEN REVIEWED YET!

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

All Jobs Finished; status = PASSED, However Inspection must be performed before merge can occur...

@bartgol bartgol requested a review from jgfouca April 8, 2022 02:26
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

Status Flag 'Pre-Merge Inspection' - SUCCESS: The last commit to this Pull Request has been INSPECTED AND APPROVED by [ jgfouca ]!

@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

Status Flag 'Pull Request AutoTester' - Pull Request will be Automerged

@E3SM-Bot E3SM-Bot merged commit 865b3a3 into master Apr 8, 2022
@E3SM-Bot
Copy link
Collaborator

E3SM-Bot commented Apr 8, 2022

Merge on Pull Request# 1520: IS A SUCCESS - Pull Request successfully merged

@E3SM-Bot E3SM-Bot deleted the bartgol/logging-support branch April 8, 2022 18:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jgfouca jgfouca jgfouca approved these changes

@AaronDonahue AaronDonahue Awaiting requested review from AaronDonahue

Assignees

@bartgol bartgol

Labels
enhancement New feature or request SCREAMv1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

SCREAM log messages appear in e3sm.log instead of atm.log
5 participants
@bartgol @E3SM-Bot @jeff-cohere @PeterCaldwell @jgfouca