Skip to content

Commit

Permalink
Merge Pull Request #13734 from trilinos/Trilinos/master_merge_2025011…
Browse files Browse the repository at this point in the history
…7_175821

Automatically Merged using Trilinos Master Merge AutoTester
PR Title: b'Trilinos Master Merge PR Generator: Auto PR created to promote from master_merge_20250117_175821 branch to master'
PR Author: trilinos-autotester
  • Loading branch information
trilinos-autotester authored Jan 18, 2025
2 parents 393d832 + f50c71c commit e0e55eb
Show file tree
Hide file tree
Showing 62 changed files with 3,449 additions and 290 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/AT2.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
should_skip: ${{ steps.skip_check.outputs.should_skip }}
steps:
- id: skip_check
uses: fkirc/skip-duplicate-actions@v5
uses: fkirc/skip-duplicate-actions@f75f66ce1886f00957d99748a42c724f4330bdcf # v5.3.1
with:
skip_after_successful_duplicate: 'true'

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/clang_format.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
- run: git diff HEAD > format_patch.txt
- run: if [ "$(cat format_patch.txt)" == "" ] ; then rm format_patch.txt ; else cat format_patch.txt; fi

- uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
- uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
id: upload-artf
if: ${{ hashFiles('format_patch.txt') != '' }}
with:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2

- name: Initialize CodeQL
uses: github/codeql-action/init@48ab28a6f5dbc2a99bf1e0131198dd8f1df78169 # v3.28.0
uses: github/codeql-action/init@b6a472f63d85b9c78a3ac5e89422239fc15e9b3c # v3.28.1
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
Expand Down Expand Up @@ -108,6 +108,6 @@ jobs:
ninja -j 16
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@48ab28a6f5dbc2a99bf1e0131198dd8f1df78169 # v3.28.0
uses: github/codeql-action/analyze@b6a472f63d85b9c78a3ac5e89422239fc15e9b3c # v3.28.1
with:
category: "/language:${{matrix.language}}"
2 changes: 1 addition & 1 deletion .github/workflows/dependency-review.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Harden Runner
uses: step-security/harden-runner@0080882f6c36860b6ba35c610c98ce87d4e2f26f # v2.10.2
uses: step-security/harden-runner@c95a14d0e5bab51a9f56296a4eb0e416910cd350 # v2.10.3
with:
egress-policy: audit

Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/scorecards.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,14 +58,14 @@ jobs:
# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
# format to the repository Actions tab.
- name: "Upload artifact"
uses: actions/upload-artifact@6f51ac03b9356f520e9adb1b1b7802705f340c2b # v4.5.0
uses: actions/upload-artifact@65c4c4a1ddee5b72f698fdd19549f0f0fb45cf08 # v4.6.0
with:
name: SARIF file
path: results.sarif
retention-days: 5

# Upload the results to GitHub's code scanning dashboard.
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@48ab28a6f5dbc2a99bf1e0131198dd8f1df78169 # v3.28.0
uses: github/codeql-action/upload-sarif@b6a472f63d85b9c78a3ac5e89422239fc15e9b3c # v3.28.1
with:
sarif_file: results.sarif
3 changes: 3 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,9 @@

[![OpenSSF Scorecard](https://api.securityscorecards.dev/projects/github.com/trilinos/Trilinos/badge)](https://securityscorecards.dev/viewer/?uri=github.com/trilinos/Trilinos)
[![OpenSSF Best Practices](https://www.bestpractices.dev/projects/9452/badge)](https://www.bestpractices.dev/projects/9452)
[![CodeFactor](https://www.codefactor.io/repository/github/trilinos/trilinos/badge)](https://www.codefactor.io/repository/github/trilinos/trilinos)
![GitHub contributors](https://img.shields.io/github/contributors/trilinos/Trilinos)
![GitHub commit activity](https://img.shields.io/github/commit-activity/w/trilinos/Trilinos)


The Trilinos Project is an effort to develop algorithms and enabling
Expand Down
65 changes: 8 additions & 57 deletions cmake/SimpleTesting/README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,3 @@
CMake File Structure
====================
This CMake structure attempts to capture the structure of the original
`TFW_testing_single_configure_prototype` that is used by the current / old Trilinos
Pull-Request (PR) framework.

**Note**: _Trilinos uses the **`pull_request_changes`** branch from this repository_.

Specifically, this new configuration will replace the existing `simple_testing.cmake`
script.

This diagram describes the current structure of the CMake files and how they
interact with each other.
<center>
<img src="img/CMake-structure.png" alt="CMake Structure Diagram" height="600"/>
</center>

CMake Files
-----------

Expand All @@ -32,42 +15,6 @@ CMake Files
The _guarded_ files use the CMake command [`include_guard()`][1] which should prevent that file
from being include more than once in an include chain.

Options and Variables (`simple_testing.cmake`)
----------------------------------------------
The `simple_testing.cmake` file has a number of optional parameters that can be sent into the
CMake system via `-D<VARNAME>:BOOL=<ON|OFF>` parameters:

| Option | Required? | Default | PR Override | Purpose |
|----------------------------|:---------:|---------------------------------------------|-------------|----------------------------------|
| `build_name` | YES | N/A | YES | Sets `CTEST_BUILD_NAME` |
| `ctest_submit_retry_count` | NO | 5 | | |
| `ctest_submit_retry_delay` | NO | 3 | | |
| `dashboard_model` | NO | `Experimental` | YES but NO | Set to the same value as default |
| `dashboard_track` | NO | `Experimental` | YES but NO | Set to the same value as default |
| `skip_by_parts_submit` | NO | `ON` | YES | |
| `skip_clean_build_dir` | NO | `ON` | | |
| `skip_single_submit` | NO | `ON` | | |
| `skip_update_step` | NO | `OFF` | YES | |
| `skip_upload_config_files` | NO | `OFF` | | |
| `PARALLEL_LEVEL` | NO | _num cores_ | YES | |
| `TEST_PARALLEL_LEVEL` | NO | `${PARALLEL_LEVEL}` | YES | |
| `build_root` | NO | `${CTEST_SOURCE_DIRECTORY}/nightly_testing` | | |
| `build_dir` | NO | `${build_root}/${CTEST_BUILD_NAME}` | YES | The CMake build dir |
| `configure_script` | YES | N/A | YES | See note below table |
| `package_enables` | YES | N/A | YES | `packageEnables.cmake` |
| `subprojects_file` | YES | N/A | YES | `package_subproject_list.cmake` |

1. `configure_script` points to the `cmake/std/PullRequestLinux<COMPILER><COMPILER_VERSION>TestingSettings.cmake` file.
- Example: `${WORKSPACE}/Trilinos/cmake/std/PullRequestLinuxGCC8.3.0TestingSettings.cmake`

See `TrilinosPRConfigurationStandard.py`[2] for information on what options are set to something
other than the default during normal Trilinos PR operations.

Expected Operation
------------------
The expected operation of this set of files to replace the old `simple_testing.cmake` is to load
the [`ctest-driver.cmake`](ctest-driver.cmake) file in its place.


`ctest-driver.cmake` Options
----------------------------
Expand All @@ -92,20 +39,21 @@ the [`ctest-driver.cmake`](ctest-driver.cmake) file in its place.
| `build_dir` | STRING | NO | `${build_root}/${CTEST_BUILD_NAME}` | Path to the build directory. |
| `PARALLEL_LEVEL` | STRING | NO | `<num cores>` | |
| `TEST_PARALLEL_LEVEL` | STRING | NO | `${PARALLEL_LEVEL}` | |
| `SKIP_RUN_TESTS` | BOOL | NO | OFF | Skip running any tests (any tests enabled will still compile) |


1. It might worthwhile to remove `build_root` since it's only used to create `build_dir` IF `build_dir` is not passed in
via a `-D` option.
2. Related to (1), we might also change `build_dir` to be `BUILD_DIR` and pass that in.



Example CTest call from a Trilinos PR
-------------------------------------
This is an example, for reference, of how the `ctest` command is invoked in the current/old Trilinos
This is an example, for reference, of how the `ctest` command is invoked in the current Trilinos
PR test driver.
```bash
ctest \
-S simple_testing.cmake \
-S ctest-driver.cmake \
-Dsource_dir=${WORKSPACE}/Trilinos \
-Dbuild_name=PR-9495-test-Trilinos_pullrequest_gcc_8.3.0-5164 \
-Dskip_by_parts_submit=OFF \
Expand All @@ -120,6 +68,9 @@ ctest \
-Dsubprojects_file=../package_subproject_list.cmake
```

See `TrilinosPRConfigurationStandard.py`[2] for information on what options are set to something
other than the default during normal Trilinos PR operations.

Additional Notes and Pitfalls
=============================

Expand All @@ -136,4 +87,4 @@ when only documentation or perhaps the testing framework itself is modified and
not need to spend O(5 hours) for the test suite to run.

[1]: https://cmake.org/cmake/help/latest/command/include_guard.html
[2]: https://github.com/trilinos/Trilinos/blob/master/cmake/std/trilinosprhelpers/TrilinosPRConfigurationStandard.py
[2]: https://github.com/trilinos/Trilinos/blob/master/packages/framework/pr_tools/trilinosprhelpers/TrilinosPRConfigurationStandard.py
3 changes: 3 additions & 0 deletions cmake/SimpleTesting/cmake/ctest-common.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,9 @@ if( NOT DEFINED skip_upload_config_files )
set( skip_upload_config_files OFF )
endif()

if( NOT DEFINED SKIP_RUN_TESTS )
set (SKIP_RUN_TESTS OFF)
endif()

# -----------------------------------------------------------
# -- Miscellaneous Settings
Expand Down
28 changes: 17 additions & 11 deletions cmake/SimpleTesting/cmake/ctest-stage-test.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -13,20 +13,26 @@ banner("START test step")

set(STAGE_TEST_ERROR OFF)

if(CTEST_BUILD_NAME MATCHES .*_asan_.*)
set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
set(ENV{LSAN_OPTIONS} "suppressions=${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/lsan.supp")
set(ENV{LD_PRELOAD} ${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/dummy_dlclose.so)
ctest_memcheck(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
if(NOT SKIP_RUN_TESTS)
if(CTEST_BUILD_NAME MATCHES .*_asan_.*)
set(CTEST_MEMORYCHECK_TYPE "AddressSanitizer")
set(ENV{LSAN_OPTIONS} "suppressions=${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/lsan.supp")
set(ENV{LD_PRELOAD} ${CTEST_SOURCE_DIRECTORY}/packages/framework/asan_assets/dummy_dlclose.so)
ctest_memcheck(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
CAPTURE_CMAKE_ERROR captured_cmake_error
RETURN_VALUE test_error)
unset(ENV{LD_PRELOAD})
submit_by_parts( "MemCheck" )
else()
ctest_test(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
CAPTURE_CMAKE_ERROR captured_cmake_error
RETURN_VALUE test_error)
unset(ENV{LD_PRELOAD})
submit_by_parts( "MemCheck" )
submit_by_parts( "Test" )
endif()
else()
ctest_test(PARALLEL_LEVEL ${TEST_PARALLEL_LEVEL}
CAPTURE_CMAKE_ERROR captured_cmake_error
RETURN_VALUE test_error)
submit_by_parts( "Test" )
message(">>> SKIPPED RUNNING TESTS (SKIP_RUN_TESTS=${SKIP_RUN_TESTS})")
set(test_error 0)
submit_by_parts("Test")
endif()

# Print out final stage banner
Expand Down
17 changes: 16 additions & 1 deletion packages/framework/get_dependencies.sh
Original file line number Diff line number Diff line change
Expand Up @@ -36,8 +36,23 @@ function tril_genconfig_clone_or_update_repo() {
echo

if [[ -d ${sub_dir} ]] ; then
echo "STATUS: ${sub_dir}: Fetching remote repo"
cd ${sub_dir}
# Validate correct remote and abort if not correct
remote=$(git remote get-url origin)
normalized_remote=$(python3 ${script_dir}/normalize_git_repo_url.py ${remote})
normalized_git_url=$(python3 ${script_dir}/normalize_git_repo_url.py ${git_url})
if [[ "${normalized_remote}" != "${normalized_git_url}" ]] ; then
echo "ERROR: Current remote origin does not match expected!" >&2
echo " Current Remote: ${remote}" >&2
echo " Expected Remote: ${git_url}" >&2
echo "" >&2
echo "Please remove/move '$(pwd)' and re-run this script" >&2
echo "" >&2
echo "" >&2
exit 1
fi
# Update remote repo (which points to correct remote)
echo "STATUS: ${sub_dir}: Fetching remote repo"
tril_genconfig_assert_pwd_is_git_repo
cmd="git fetch"
retry_command "${cmd}"
Expand Down
24 changes: 20 additions & 4 deletions packages/framework/ini-files/config-specs.ini
Original file line number Diff line number Diff line change
Expand Up @@ -1517,16 +1517,32 @@ use PACKAGE-ENABLES|NO-EPETRA
use COMMON_SPACK_TPLS
use SEMS_CUDA

opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : OFF
# If run through PR testing, PR scripts will control whether tests
# should build and run, or ONLY build (TRILFRAME-675)
opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : ON

# Packages containing tests broken when UVM enabled
opt-set-cmake-var Amesos2_ENABLE_TESTS BOOL FORCE : OFF
opt-set-cmake-var NOX_ENABLE_TESTS BOOL FORCE : OFF
opt-set-cmake-var Panzer_ENABLE_TESTS BOOL FORCE : OFF
opt-set-cmake-var Stokhos_ENABLE_TESTS BOOL FORCE : OFF
opt-set-cmake-var TrilinosCouplings_ENABLE_TESTS BOOL FORCE : OFF

[rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all]
use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
use PACKAGE-ENABLES|ALL-NO-EPETRA
use PACKAGE-ENABLES|ALL

[rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all-no-epetra]
use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_all
use rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_uvm_deprecated-on_no-package-enables
use PACKAGE-ENABLES|ALL-NO-EPETRA

opt-set-cmake-var Trilinos_ENABLE_TESTS BOOL FORCE : ON
# Packages containing tests broken when UVM enabled
# Enabled in this configuration to be caught in Nightly
opt-set-cmake-var Amesos2_ENABLE_TESTS BOOL FORCE : ON
opt-set-cmake-var NOX_ENABLE_TESTS BOOL FORCE : ON
opt-set-cmake-var Panzer_ENABLE_TESTS BOOL FORCE : ON
opt-set-cmake-var Stokhos_ENABLE_TESTS BOOL FORCE : ON
opt-set-cmake-var TrilinosCouplings_ENABLE_TESTS BOOL FORCE : ON

[rhel8_sems-cuda-11.4.2-gnu-10.1.0-openmpi-4.1.6_release_static_Volta70_no-asan_complex_no-fpic_mpi_pt_no-rdc_no-uvm_deprecated-on_no-package-enables]
# uses sems-v2 modules
Expand Down
36 changes: 36 additions & 0 deletions packages/framework/normalize_git_repo_url.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/usr/bin/env python3

import sys

def removeStartingSubstr(inputStr, substrToRemove):
if inputStr.startswith(substrToRemove):
substrToRemoveLen = len(substrToRemove)
return inputStr[substrToRemoveLen:]
return inputStr

def removeTrailingSubstr(inputStr, substrToRemove):
if inputStr.endswith(substrToRemove):
substrToRemoveLen = len(substrToRemove)
return inputStr[:-substrToRemoveLen]
return inputStr

def normalizeGitRepoUrl(inputUrl):
url = inputUrl
url = removeStartingSubstr(url, "https://")
url = removeStartingSubstr(url, "git@")
url = removeTrailingSubstr(url, ".git")
url = url.replace(":", "/")
url = url.lower()
return url

# Main

def main():
if len(sys.argv) != 2:
print("Usage: normalize_git_repo_url.py <string>")
sys.exit(1)
inputUrl = sys.argv[1]
print(normalizeGitRepoUrl(inputUrl))

if __name__ == "__main__":
main()
5 changes: 5 additions & 0 deletions packages/framework/pr_tools/PullRequestLinuxDriver.sh
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,11 @@ then
test_cmd_options+=( "--skip-create-packageenables ")
fi

if [[ ${GENCONFIG_BUILD_NAME} == *"_uvm_"* && ${GENCONFIG_BUILD_NAME} == *"no-package-enables"* ]]
then
test_cmd_options+=( "--skip-run-tests" )
fi

test_cmd="${PYTHON_EXE:?} ${REPO_ROOT:?}/packages/framework/pr_tools/PullRequestLinuxDriverTest.py ${test_cmd_options[@]}"

# Call the script to launch the tests
Expand Down
7 changes: 7 additions & 0 deletions packages/framework/pr_tools/PullRequestLinuxDriverTest.py
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,12 @@ def parse_args():
default=False,
help="Enable dry-run mode. Script will run but not execute the build steps. Default = %(default)s")

optional.add_argument("--skip-run-tests",
dest="skip_run_tests",
action="store_true",
default=False,
help="Skip running tests in SimpleTesting test stage. Default = %(default)s")

optional.add_argument("--extra-configure-args",
dest="extra_configure_args",
action="store",
Expand Down Expand Up @@ -299,6 +305,7 @@ def parse_args():
print("| - [O] req-mem-per-core : {req_mem_per_core}".format(**vars(arguments)))
print("| - [O] test-mode : {test_mode}".format(**vars(arguments)))
print("| - [O] workspace-dir : {workspace_dir}".format(**vars(arguments)))
print("| - [O] skip-run-tests : {skip_run_tests}".format(**vars(arguments)))
print("| - [O] extra_configure_args : {extra_configure_args}".format(**vars(arguments)))
print("| - [O] dashboard_build_name : {dashboard_build_name}".format(**vars(arguments)))
print("| - [O] use_explicit_cachefile : {use_explicit_cachefile}".format(**vars(arguments)))
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -352,6 +352,15 @@ def arg_ccache_enable(self):
"""Is ccache enabled?"""
return self.args.ccache_enable

@property
def arg_skip_run_tests(self):
"""
Control whether tests should run for this build. Used for cases
where resources are limited such that you choose to only compile tests
but skip running them.
"""
return self.args.skip_run_tests

# --------------------
# P R O P E R T I E S
# --------------------
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@ def execute_test(self):
f"-Dsubprojects_file:FILEPATH={self.arg_filename_subprojects}",
f"-DCTEST_DROP_SITE:STRING={self.arg_ctest_drop_site}",
"-DUSE_EXPLICIT_TRILINOS_CACHEFILE:BOOL=" + ("ON" if self.arg_use_explicit_cachefile else "OFF"),
"-DSKIP_RUN_TESTS:BOOL=" + ("ON" if self.arg_skip_run_tests else "OFF"),
]

if self.arg_extra_configure_args:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -175,7 +175,8 @@ def dummy_args(self):
ccache_enable=False,
dry_run = False,
use_explicit_cachefile = False,
extra_configure_args = ""
extra_configure_args = "",
skip_run_tests = False
)
return output

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ def setUp(self):
filename_packageenables='../packageEnables.cmake',
filename_subprojects='../package_subproject_list.cmake',
skip_create_packageenables=False,
skip_run_tests=False,
test_mode='standard',
req_mem_per_core=3.0,
max_cores_allowed=12,
Expand Down
Loading

0 comments on commit e0e55eb

Please sign in to comment.