From 1988ada3a5111f5190c4f3abaf999d4cfc0bdd2b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Fri, 10 Oct 2025 18:25:18 +0200 Subject: [PATCH 01/13] dasharo-performance-parallelable: Add proof of concept MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit It's a proof of concept for running test cases in parallel, with common execution steps. This apporach, while might look complicated and non-axiomatic, can allow for devastating improvements in test suite execution times. By using the supported test cases dict, _CANARY_ and _PSEUDO_ test cases the actual test scope can be determined, then all the tests can be performed in parallel in any way we find reasonable. Then the actual test cases can access suite variables created by the _PSEUDO_ test cases to determine whether they PASS or not. Signed-off-by: Filip Gołaś --- .../measurements.robot | 78 +++++++++++++++++++ 1 file changed, 78 insertions(+) create mode 100644 dasharo-performance-parallelable/measurements.robot diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot new file mode 100644 index 0000000000..f05031d6b6 --- /dev/null +++ b/dasharo-performance-parallelable/measurements.robot @@ -0,0 +1,78 @@ +*** Settings *** +Library Collections +Library OperatingSystem +Library Process +Library String +Library Telnet timeout=20 seconds connection_timeout=120 seconds +Library SSHLibrary timeout=90 seconds +Library RequestsLibrary +# TODO: maybe have a single file to include if we need to include the same +# stuff in all test cases +Resource ../variables.robot +Resource ../keywords.robot +Resource ../keys.robot +Resource ../keys-and-keywords/heads-keywords.robot +Resource ../lib/performance/cpu.robot +# TODO: +# - document which setup/teardown keywords to use and what are they doing +# - go threough them and make sure they are doing what the name suggest (not +# exactly the case right now) + + +*** Variables *** +&{TESTS_TO_RUN}= &{EMPTY} +${CPU_FREQUENCY_MEASURE}= ${TRUE} +${TESTS_IN_UBUNTU_SUPPORT}= ${FALSE} +@{TESTED_LINUX_DISTROS}= 202 201 + + +*** Test Cases *** +_CANARY_CPF001.201 + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... CPF001.201 + ... not ${CPU_FREQUENCY_MEASURE} + ... frequency measure not supported + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... CPF001.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... CPF001.201 + ... '201' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + +_PSEUDO_Background Measurements Data Init (no load) (Ubuntu) + Log Doing some init WARN + +CPF001.201 CPU not stuck on initial frequency (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU does not + ... stuck on the initial frequency after booting into the OS. + ... Previous IDs: CPF001.001 + Skip If Not Supported CPF001.201 ${TESTS_TO_RUN} + + +*** Keywords *** +Add Test Case Skip Condition + [Arguments] ${tests_dict} ${test_id} ${condition} ${skip_reason} + ${current_details}= Get From Dictionary ${tests_dict} ${test_id} default=${FALSE} + IF not ${current_details} + VAR &{support_details}= test=${test_id} run=${TRUE} reason=supported + ELSE + VAR ${support_details}= ${current_details} + END + ${run}= Get From Dictionary ${support_details} run + IF ${run} and (${condition}) + Set To Dictionary ${support_details} run=${FALSE} reason=${skip_reason} + END + Set To Dictionary ${tests_dict} ${test_id}=${support_details} + +Skip If Not Supported + [Arguments] ${test_id} ${tests_dict} + &{support_details}= Get From Dictionary ${tests_dict} ${test_id} + ${run}= Get From Dictionary ${support_details} run + ${reason}= Get From Dictionary ${support_details} reason + ${not_run}= Evaluate not ${run} + Skip If ${not_run} ${reason} From 1fa3ee0892c11263a51e52714832e406c82dc92e Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 15 Oct 2025 08:35:52 +0200 Subject: [PATCH 02/13] measurements.robot: Add CPF001.201 test case to POC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 142 ++++++++++++++++-- 1 file changed, 130 insertions(+), 12 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index f05031d6b6..128b0e3f69 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -13,45 +13,121 @@ Resource ../keywords.robot Resource ../keys.robot Resource ../keys-and-keywords/heads-keywords.robot Resource ../lib/performance/cpu.robot +Resource ../lib/sensors/sensors.robot # TODO: # - document which setup/teardown keywords to use and what are they doing # - go threough them and make sure they are doing what the name suggest (not # exactly the case right now) +Suite Setup Run Keywords Prepare Test Suite *** Variables *** &{TESTS_TO_RUN}= &{EMPTY} ${CPU_FREQUENCY_MEASURE}= ${TRUE} -${TESTS_IN_UBUNTU_SUPPORT}= ${FALSE} +${TESTS_IN_UBUNTU_SUPPORT}= ${TRUE} @{TESTED_LINUX_DISTROS}= 202 201 *** Test Cases *** + +# Tests that can be performed immediately _CANARY_CPF001.201 + VAR ${TEST_ID}= CPF001.201 Add Test Case Skip Condition ... ${TESTS_TO_RUN} - ... CPF001.201 + ... ${TEST_ID} ... not ${CPU_FREQUENCY_MEASURE} ... frequency measure not supported Add Test Case Skip Condition ... ${TESTS_TO_RUN} - ... CPF001.201 + ... ${TEST_ID} ... not ${TESTS_IN_UBUNTU_SUPPORT} ... tests in Ubuntu not supported Add Test Case Skip Condition ... ${TESTS_TO_RUN} - ... CPF001.201 + ... ${TEST_ID} ... '201' not in ${TESTED_LINUX_DISTROS} ... Ubuntu not in tested distros + Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} + +# _CANARY_CPF005.201 CPU runs on expected frequency (Ubuntu) +# VAR ${TEST_ID}= CPF005.201 +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... not ${CPU_FREQUENCY_MEASURE} +# ... frequency measure not supported +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... not ${TESTS_IN_UBUNTU_SUPPORT} +# ... tests in Ubuntu not supported +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... '201' not in ${TESTED_LINUX_DISTROS} +# ... Ubuntu not in tested distros +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... ${LAPTOP_PLATFORM} +# ... The Platform is a Laptop + +# _CANARY_CPT001.201 CPU temperature without load (Ubuntu) +# VAR ${TEST_ID}= CPT001.201 +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... not ${TESTS_IN_UBUNTU_SUPPORT} +# ... tests in Ubuntu not supported +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... ${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} +# ... Ubuntu not in tested distros +# Add Test Case Skip Condition +# ... ${TESTS_TO_RUN} +# ... ${TEST_ID} +# ... ${LAPTOP_PLATFORM} +# ... The Platform is a Laptop + + +_GATHER_Background Measurements Data Init (no load) (Ubuntu) + ${prepare_sensors}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] + IF ${prepare_sensors} Prepare Sensors -_PSEUDO_Background Measurements Data Init (no load) (Ubuntu) - Log Doing some init WARN +_GATHER_Background Measurements Immediate (no load) (Ubuntu) + ${will_any_be_run}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] + Skip If not ${will_any_be_run} + + Power On + Boot System Or From Connected Disk ${ENV_ID_UBUNTU} + Login To Linux + Switch To Root User + + # CPF001.201 steps + VAR ${test_id}= CPF001.201 + ${check_frequency}= Will Test Be Run ${TESTS_TO_RUN} ${test_id} + IF ${check_frequency} + Sleep 10s + @{frequencies}= Get CPU Frequencies In Ubuntu + Set Test Outputs ${TESTS_TO_RUN} ${test_id} ${frequencies} + END CPF001.201 CPU not stuck on initial frequency (Ubuntu) [Documentation] This test aims to verify whether the mounted CPU does not ... stuck on the initial frequency after booting into the OS. ... Previous IDs: CPF001.001 - Skip If Not Supported CPF001.201 ${TESTS_TO_RUN} + VAR ${test_id}= CPF001.201 + Skip If Not Supported ${test_id} ${TESTS_TO_RUN} + @{frequencies}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + ${first_frequency}= Get From List ${frequencies} 0 + FOR ${frequency} IN @{frequencies} + IF ${frequency} != ${INITIAL_CPU_FREQUENCY} + Pass Execution CPU does not stuck on initial frequency + END + END + Fail CPU stuck on initial frequency: ${INITIAL_CPU_FREQUENCY} *** Keywords *** @@ -71,8 +147,50 @@ Add Test Case Skip Condition Skip If Not Supported [Arguments] ${test_id} ${tests_dict} - &{support_details}= Get From Dictionary ${tests_dict} ${test_id} - ${run}= Get From Dictionary ${support_details} run - ${reason}= Get From Dictionary ${support_details} reason - ${not_run}= Evaluate not ${run} - Skip If ${not_run} ${reason} + ${will_be_run}= Will Test Be Run ${tests_dict} ${test_id} + Skip If not $will_be_run + +Will Test Be Run + [Arguments] ${tests_dict} ${test_id} + ${test}= Get From Dictionary ${tests_dict} ${test_id} ${None} + IF $test is ${None} + Log Test ${test_id} not found + RETURN ${FALSE} + END + ${value}= Get From Dictionary ${test} run ${FALSE} + RETURN ${value} + + +Get Test Value + [Arguments] ${tests_dict} ${test} ${key} + ${test}= Get From Dictionary ${tests_dict} ${test} + ${value}= Get From Dictionary ${test} ${key} + RETURN ${value} + +Get Test Outputs + [Arguments] ${tests_dict} ${test} + ${outputs}= Get Test Value ${tests_dict} ${test} outputs + RETURN ${outputs} + +Set Test Value + [Arguments] ${tests_dict} ${test_id} ${key} ${value} + ${test}= Get From Dictionary ${tests_dict} ${test_id} + Set To Dictionary ${test} ${key}=${value} + Set To Dictionary ${tests_dict} ${test_id}=${test} + +Set Test Outputs + [Arguments] ${tests_dict} ${test} ${value} + Set Test Value ${tests_dict} ${test} outputs ${value} + +*** Keywords *** +Will Test Be Run Regex + [Arguments] ${dict} ${pattern} + FOR ${key} IN @{dict.keys()} + ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re + IF $match is not ${None} + ${id}= Evaluate $match.string + ${run}= Will Test Be Run ${TESTS_TO_RUN} ${id} + IF ${run} RETURN ${TRUE} + END + END + RETURN ${FALSE} \ No newline at end of file From 3eb3d7e8c793de2eb318ea73571bba9cc3aca51f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 15 Oct 2025 10:58:14 +0200 Subject: [PATCH 03/13] measurements.robot: Add CPF005 and CPT001 test case to POC, parallelism MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 199 ++++++++++++++---- 1 file changed, 156 insertions(+), 43 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 128b0e3f69..2244660d6b 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -6,6 +6,7 @@ Library String Library Telnet timeout=20 seconds connection_timeout=120 seconds Library SSHLibrary timeout=90 seconds Library RequestsLibrary +Library DateTime # TODO: maybe have a single file to include if we need to include the same # stuff in all test cases Resource ../variables.robot @@ -19,7 +20,9 @@ Resource ../lib/sensors/sensors.robot # - go threough them and make sure they are doing what the name suggest (not # exactly the case right now) -Suite Setup Run Keywords Prepare Test Suite +Suite Setup Run Keywords +... Prepare Test Suite AND +... Check Power Supply *** Variables *** &{TESTS_TO_RUN}= &{EMPTY} @@ -31,7 +34,7 @@ ${TESTS_IN_UBUNTU_SUPPORT}= ${TRUE} *** Test Cases *** # Tests that can be performed immediately -_CANARY_CPF001.201 +_CANARY_CPF001.201 CPU not stuck on initial frequency (Ubuntu) VAR ${TEST_ID}= CPF001.201 Add Test Case Skip Condition ... ${TESTS_TO_RUN} @@ -50,54 +53,62 @@ _CANARY_CPF001.201 ... Ubuntu not in tested distros Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} -# _CANARY_CPF005.201 CPU runs on expected frequency (Ubuntu) -# VAR ${TEST_ID}= CPF005.201 -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... not ${CPU_FREQUENCY_MEASURE} -# ... frequency measure not supported -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... not ${TESTS_IN_UBUNTU_SUPPORT} -# ... tests in Ubuntu not supported -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... '201' not in ${TESTED_LINUX_DISTROS} -# ... Ubuntu not in tested distros -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... ${LAPTOP_PLATFORM} -# ... The Platform is a Laptop - -# _CANARY_CPT001.201 CPU temperature without load (Ubuntu) -# VAR ${TEST_ID}= CPT001.201 -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... not ${TESTS_IN_UBUNTU_SUPPORT} -# ... tests in Ubuntu not supported -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... ${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} -# ... Ubuntu not in tested distros -# Add Test Case Skip Condition -# ... ${TESTS_TO_RUN} -# ... ${TEST_ID} -# ... ${LAPTOP_PLATFORM} -# ... The Platform is a Laptop +_CANARY_CPF005.201 CPU runs on expected frequency (Ubuntu) + VAR ${TEST_ID}= CPF005.201 + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... not ${CPU_FREQUENCY_MEASURE} + ... frequency measure not supported + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... '201' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... ${LAPTOP_PLATFORM} + ... The Platform is a Laptop + Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} + +_CANARY_CPT001.201 CPU temperature without load (Ubuntu) + VAR ${TEST_ID}= CPT001.201 + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + Add Test Case Skip Condition + ... ${TESTS_TO_RUN} + ... ${TEST_ID} + ... ${LAPTOP_PLATFORM} + ... The Platform is a Laptop + Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} +############################################ +# Tests that can be done immediately # +############################################ _GATHER_Background Measurements Data Init (no load) (Ubuntu) ${prepare_sensors}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] IF ${prepare_sensors} Prepare Sensors + ${check_psu}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] + IF ${check_psu} Check Power Supply _GATHER_Background Measurements Immediate (no load) (Ubuntu) - ${will_any_be_run}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] + # immediately skip if no tests want these measurements + ${will_any_be_run}= Will Test Be Run Regex ${TESTS_TO_RUN} CPF001.201 Skip If not ${will_any_be_run} Power On @@ -129,8 +140,110 @@ CPF001.201 CPU not stuck on initial frequency (Ubuntu) END Fail CPU stuck on initial frequency: ${INITIAL_CPU_FREQUENCY} +############################################################################# +# Tests that gather measurements on Ubuntu, no load, n/a power source # +############################################################################# +_GATHER_Background Measurements (no load) (Ubuntu) + ${will_any_be_run}= Will Test Be Run Regex + ... ${TESTS_TO_RUN} + ... (CPF005)|(CPT001)|(STB001).201 + Skip If not ${will_any_be_run} + + Power On + Boot System Or From Connected Disk ${ENV_ID_UBUNTU} + Login To Linux + Switch To Root User + + # CPF001.201 steps + ${gather_temps}= Will Test Be Run Regex ${TESTS_TO_RUN} CPT + ${gather_freqs}= Will Test Be Run Regex ${TESTS_TO_RUN} CPF + ${measure_stab}= Will Test Be Run Regex ${TESTS_TO_RUN} STB + + VAR @{temp_list}= @{EMPTY} + VAR @{freq_list}= @{EMPTY} + VAR @{stab_list}= @{EMPTY} + + # TODO temporary debug values + VAR ${FREQUENCY_TEST_MEASURE_INTERVAL}= 1 + VAR ${TEMPERATURE_TEST_MEASURE_INTERVAL}= 1 + VAR ${TEMPERATURE_TEST_DURATION}= 5 + VAR ${FREQUENCY_TEST_DURATION}= 5 + VAR ${STABILITY_TEST_DURATION}= 0 + + ${next_temp_time}= Set Variable If ${gather_temps} ${TEMPERATURE_TEST_MEASURE_INTERVAL} 999999 + ${next_stab_time}= Set Variable If ${gather_temps} ${STABILITY_TEST_MEASURE_INTERVAL} 999999 + ${next_freq_time}= Set Variable If ${gather_temps} ${FREQUENCY_TEST_MEASURE_INTERVAL} 999999 + + ${start}= DateTime.Get Current Date + ${timer}= Evaluate 0 + VAR ${longest_duration}= max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}, ${STABILITY_TEST_DURATION}) + + WHILE ${timer} < ${longest_duration} + ${now}= Get Current Date + ${timer}= Subtract Date From Date ${now} ${start} + + IF ${gather_temps} and ${TEMPERATURE_TEST_DURATION} >= ${timer} >= ${next_temp_time} + ${temperature}= Get CPU Temperature + ${next_temp_time}= Evaluate ${timer} + ${TEMPERATURE_TEST_MEASURE_INTERVAL} + Append To List ${temp_list} ${temperature} + Log To Console ${timer}s: Temperature: ${temperature} + END + + IF ${gather_freqs} and ${FREQUENCY_TEST_DURATION} >= ${timer} >= ${next_freq_time} + ${freqs}= Get CPU Frequencies In Ubuntu + ${next_freq_time}= Evaluate ${timer} + ${FREQUENCY_TEST_MEASURE_INTERVAL} + Append To List ${freq_list} ${freqs} + Log To Console ${timer}s: Frequencies: ${freqs} + END + # ... Log To Console Stability check at ${elapsed}s + + # Run Keyword If ${elapsed} % ${FREQUENCY_TEST_MEASURE_INTERVAL} < 0.1 and ${elapsed} <= ${FREQUENCY_TEST_DURATION} + # ... Log To Console Frequency check at ${elapsed}s + + ${time_to_next_interval}= Evaluate min(${next_temp_time}, ${next_stab_time}, ${next_freq_time}) + Sleep ${time_to_next_interval} + END + + Set Test Outputs ${TESTS_TO_RUN} CPT001.201 ${temp_list} + Set Test Outputs ${TESTS_TO_RUN} CPF005.201 ${freq_list} + +CPT001.201 CPU temperature without load (Ubuntu) + VAR ${test_id}= CPT001.201 + ${temps}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + ${sum}= Evaluate 0 + ${len}= Get Length ${temps} + FOR ${temp} IN @{temps} + ${sum}= Evaluate ${sum} + ${temp} + END + ${avg}= Evaluate ${sum} / ${len} + ${min}= Evaluate min($temps) + ${max}= Evaluate max($temps) + Log To Console Average temp: ${avg} + Log To Console Min temp: ${min} + Log To Console Max temp: ${max} + Log To Console Test threshold of CPU temp: ${MAX_CPU_TEMP}°C + Should Be True ${avg} < ${MAX_CPU_TEMP} Average is higher than threshold + +CPF005.201 CPU runs on expected frequency (Ubuntu) + VAR ${test_id}= CPF005.201 + ${freqs}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + ${cpu_max_frequency_tol}= Evaluate ${CPU_MAX_FREQUENCY} * 1.125 + ${cpu_min_frequency_tol}= Evaluate ${CPU_MIN_FREQUENCY} * 0.875 + FOR ${freqs_cores} IN @{freqs} + FOR ${core} IN @{freqs_cores} + ${in_range}= Evaluate ${cpu_min_frequency_tol} <= ${core} <= ${cpu_max_frequency_tol} + Should Be True ${in_range} Encountered invalid frequency: ${in_range} MHz + END + END *** Keywords *** +Wait Until Time + [Arguments] ${target} + ${now}= Get Current Date result_format=%Y-%m-%d %H:%M:%S.%f + ${diff}= Subtract Date From Date ${target} ${now} result_format=number + ${sleep}= Evaluate max(min(${diff}, 0.1), 0.01) + Sleep ${sleep} + Add Test Case Skip Condition [Arguments] ${tests_dict} ${test_id} ${condition} ${skip_reason} ${current_details}= Get From Dictionary ${tests_dict} ${test_id} default=${FALSE} From 261819fbf3c8f4ada64aebf97521cd377ab86138 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Mon, 20 Oct 2025 17:42:06 +0200 Subject: [PATCH 04/13] scripts/lib/robot.sh: Give matched tests and tags to robot as vars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- scripts/lib/robot.sh | 111 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 107 insertions(+), 4 deletions(-) diff --git a/scripts/lib/robot.sh b/scripts/lib/robot.sh index ee0eeacf32..5d056e30ae 100644 --- a/scripts/lib/robot.sh +++ b/scripts/lib/robot.sh @@ -75,6 +75,105 @@ check_dirty_tree() { fi } +bash_list_to_python_list_string() { + list=("${@:1}") + if ((${#list[@]})) || [[ -z ${list[0]} ]]; then + printf -v python_list '%s,' "${list[@]}" + python_list=${python_list%,} + echo "[\"${python_list//,/\",\"}\"]" + else + echo "[]" + fi +} + +test_matches_pattern() { + # Robot Framework 7.3 + # Select tests by name or by long name containing also parent suite name like + # Parent. Test. Name is case and space insensitive and it can also be a simple + # pattern where * matches anything, ? matches any single character, and + # [chars] matches one character in brackets. + test_name="$1" + pattern="$2" + # remove uppercase, spacebars to be case and space insensitive + # remove quotes if they got here, not needed when spaces are removed + norm_name=$(echo "$test_name" | tr '[:upper:]' '[:lower:]' | tr -d ' ' | tr -d '"') + # normalize pattern the same way + norm_pat=$(echo "$pattern" | tr '[:upper:]' '[:lower:]' | tr -d ' ' | tr -d '"') + # treat robot pattern as shell glob - [], *, ? work identically in POSIX + if [[ "$norm_name" == $norm_pat ]]; then + return 0 + fi + return 1 +} + +get_matched_test_cases() { + test_name=$1 # file or directory + IFS='-' read -ra robot_args <<< "$2" # like '-v 123 -t "*test1*" -i basic' + t_args=() + all_test_cases=() + test_cases_to_execute=() + + # Scan for all -i and -t parameters and save them + for arg in "${robot_args[@]}"; do + arg=$(echo "$arg" | xargs) # trim + case "$arg" in + t\ *) t_args+=("$(echo "$arg" | cut -d' ' -f2-)") ;; + *) ;; + esac + done + + # Find all test cases to run according to given module/suite file + if [ -d "$test_name" ]; then + while IFS= read -r file; do + while IFS= read -r line; do + all_test_cases+=("$line") + done < <(grep -hE '^[A-Z]{3,8}[0-9]{3}\.[0-9]{3}' "$file") + done < <(find "$test_name" -type f -name "*.robot") + elif [ -f "$test_name" ]; then + while IFS= read -r line; do + all_test_cases+=("$line") + done < <(grep -hE '^[A-Z]{3,8}[0-9]{3}\.[0-9]{3}' "$test_name") + else + echo "Error invalid file or directory $test_name" >&2 + return 1 + fi + + # filter test cases using -t parameter + if [ -n "$t_args" ]; then + while IFS= read -r case; do + for filter in "${t_args[@]}"; do + if test_matches_pattern "$case" "$filter"; then + test_cases_to_execute+=("$case") + break + fi + done + done < <(printf "%s\n" "${all_test_cases[@]}") + else + test_cases_to_execute=("${all_test_cases[@]}") + fi + + # leave only test IDs of matched test cases + mapfile -t test_cases_to_execute < <( + printf "%s\n" "${test_cases_to_execute[@]}" | awk '{print $1}' + ) + + echo "$(bash_list_to_python_list_string "${test_cases_to_execute[@]}")" +} + +get_test_tags() { + IFS='-' read -ra robot_args <<< "$2" # like '-v 123 -t "*test1*" -i basic' + tags=() + # Scan for all -i parameters and save them + for arg in "${robot_args[@]}"; do + arg=$(echo "$arg" | xargs) # trim + case "$arg" in + i\ *) tags+=("$(echo "$arg" | cut -d' ' -f2-)") ;; + *) ;; + esac + done + echo "$(bash_list_to_python_list_string "${tags[@]}")" +} + execute_robot() { # _test_path can be either # - path to directory containing a set of .robot files @@ -216,6 +315,11 @@ execute_robot() { echo "Logs will be saved at ${_logs_dir}" echo "Watch \"${_debug_file}\" to monitor the progress of the test" + _test_cases=$(get_matched_test_cases "$_test_name" "${_robot_args[*]}") + [[ -n $_test_cases ]] && _test_cases="-v TEST_CASES:'$_test_cases'" + _test_tags=$(get_test_tags "${_robot_args[*]}") + [[ -n $_test_tags ]] && _test_tags="-v TEST_TAGS:'$_test_tags'" + command=" robot -L TRACE \ -l ${_log_file} \ @@ -231,13 +335,12 @@ execute_robot() { ${installed_dut_option} \ ${extra_options} \ ${_robot_args[*]} \ - ${_test_name} + ${_test_cases} \ + ${_test_tags} \ + ${_test_name} \ " # echo "$command" eval "$command" - if [[ $? -ne 0 ]]; then - overall_rc=1 - fi done return $overall_rc } From 823bca197a41eb3379e21cff4c29dda9f21e34f7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Mon, 20 Oct 2025 18:14:03 +0200 Subject: [PATCH 05/13] measurements.robot: Use wrapper list of tests instead of canaries MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 281 +++++------------- lib/parallel-testing.robot | 114 +++++++ 2 files changed, 192 insertions(+), 203 deletions(-) create mode 100644 lib/parallel-testing.robot diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 2244660d6b..645a4f5ad0 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -1,114 +1,31 @@ *** Settings *** -Library Collections -Library OperatingSystem -Library Process -Library String -Library Telnet timeout=20 seconds connection_timeout=120 seconds -Library SSHLibrary timeout=90 seconds -Library RequestsLibrary -Library DateTime -# TODO: maybe have a single file to include if we need to include the same -# stuff in all test cases -Resource ../variables.robot -Resource ../keywords.robot -Resource ../keys.robot -Resource ../keys-and-keywords/heads-keywords.robot -Resource ../lib/performance/cpu.robot -Resource ../lib/sensors/sensors.robot -# TODO: -# - document which setup/teardown keywords to use and what are they doing -# - go threough them and make sure they are doing what the name suggest (not -# exactly the case right now) - -Suite Setup Run Keywords -... Prepare Test Suite AND -... Check Power Supply - -*** Variables *** -&{TESTS_TO_RUN}= &{EMPTY} -${CPU_FREQUENCY_MEASURE}= ${TRUE} -${TESTS_IN_UBUNTU_SUPPORT}= ${TRUE} -@{TESTED_LINUX_DISTROS}= 202 201 +Library Collections +Library DateTime +Library String +Library Telnet timeout=20 seconds connection_timeout=120 seconds +Library SSHLibrary timeout=90 seconds + +Resource ../variables.robot +Resource ../keywords.robot +Resource ../lib/performance/cpu.robot +Resource ../lib/sensors/sensors.robot +Resource ../lib/parallel-testing.robot + +Suite Setup Run Keywords +... Prepare Test Suite +... AND Run Keyword If "CPT" in " ".join($TEST_CASES) or "CPF" in " ".join($TEST_CASES) +... Check Power Supply +... AND Init Parallel Testing +... AND Prepare Parallel Test Suite *** Test Cases *** - -# Tests that can be performed immediately -_CANARY_CPF001.201 CPU not stuck on initial frequency (Ubuntu) - VAR ${TEST_ID}= CPF001.201 - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... not ${CPU_FREQUENCY_MEASURE} - ... frequency measure not supported - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... not ${TESTS_IN_UBUNTU_SUPPORT} - ... tests in Ubuntu not supported - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... '201' not in ${TESTED_LINUX_DISTROS} - ... Ubuntu not in tested distros - Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} - -_CANARY_CPF005.201 CPU runs on expected frequency (Ubuntu) - VAR ${TEST_ID}= CPF005.201 - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... not ${CPU_FREQUENCY_MEASURE} - ... frequency measure not supported - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... not ${TESTS_IN_UBUNTU_SUPPORT} - ... tests in Ubuntu not supported - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... '201' not in ${TESTED_LINUX_DISTROS} - ... Ubuntu not in tested distros - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... ${LAPTOP_PLATFORM} - ... The Platform is a Laptop - Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} - -_CANARY_CPT001.201 CPU temperature without load (Ubuntu) - VAR ${TEST_ID}= CPT001.201 - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... not ${TESTS_IN_UBUNTU_SUPPORT} - ... tests in Ubuntu not supported - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} - ... Ubuntu not in tested distros - Add Test Case Skip Condition - ... ${TESTS_TO_RUN} - ... ${TEST_ID} - ... ${LAPTOP_PLATFORM} - ... The Platform is a Laptop - Skip If Not Supported ${TEST_ID} ${TESTS_TO_RUN} - - ############################################ # Tests that can be done immediately # ############################################ -_GATHER_Background Measurements Data Init (no load) (Ubuntu) - ${prepare_sensors}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] - IF ${prepare_sensors} Prepare Sensors - ${check_psu}= Will Test Be Run Regex ${TESTS_TO_RUN} CP[TF] - IF ${check_psu} Check Power Supply - -_GATHER_Background Measurements Immediate (no load) (Ubuntu) +_PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # immediately skip if no tests want these measurements - ${will_any_be_run}= Will Test Be Run Regex ${TESTS_TO_RUN} CPF001.201 + ${will_any_be_run}= Will Parallel Test Be Run Regex CPF001.201 Skip If not ${will_any_be_run} Power On @@ -117,21 +34,21 @@ _GATHER_Background Measurements Immediate (no load) (Ubuntu) Switch To Root User # CPF001.201 steps - VAR ${test_id}= CPF001.201 - ${check_frequency}= Will Test Be Run ${TESTS_TO_RUN} ${test_id} + VAR ${PARALLEL_TEST_ID}= CPF001.201 + ${check_frequency}= Will Parallel Test Be Run ${PARALLEL_TEST_ID} IF ${check_frequency} Sleep 10s @{frequencies}= Get CPU Frequencies In Ubuntu - Set Test Outputs ${TESTS_TO_RUN} ${test_id} ${frequencies} + Set Parallel Test Outputs ${PARALLEL_TEST_ID} ${frequencies} END CPF001.201 CPU not stuck on initial frequency (Ubuntu) [Documentation] This test aims to verify whether the mounted CPU does not ... stuck on the initial frequency after booting into the OS. ... Previous IDs: CPF001.001 - VAR ${test_id}= CPF001.201 - Skip If Not Supported ${test_id} ${TESTS_TO_RUN} - @{frequencies}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + VAR ${PARALLEL_TEST_ID}= CPF001.201 + Skip If Parallel Test Not Supported ${PARALLEL_TEST_ID} + @{frequencies}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} ${first_frequency}= Get From List ${frequencies} 0 FOR ${frequency} IN @{frequencies} IF ${frequency} != ${INITIAL_CPU_FREQUENCY} @@ -143,9 +60,8 @@ CPF001.201 CPU not stuck on initial frequency (Ubuntu) ############################################################################# # Tests that gather measurements on Ubuntu, no load, n/a power source # ############################################################################# -_GATHER_Background Measurements (no load) (Ubuntu) - ${will_any_be_run}= Will Test Be Run Regex - ... ${TESTS_TO_RUN} +_PARALLEL_Background Measurements (no load) (Ubuntu) + ${will_any_be_run}= Will Parallel Test Be Run Regex ... (CPF005)|(CPT001)|(STB001).201 Skip If not ${will_any_be_run} @@ -155,43 +71,45 @@ _GATHER_Background Measurements (no load) (Ubuntu) Switch To Root User # CPF001.201 steps - ${gather_temps}= Will Test Be Run Regex ${TESTS_TO_RUN} CPT - ${gather_freqs}= Will Test Be Run Regex ${TESTS_TO_RUN} CPF - ${measure_stab}= Will Test Be Run Regex ${TESTS_TO_RUN} STB + ${gather_temps}= Will Parallel Test Be Run Regex CPT + ${gather_freqs}= Will Parallel Test Be Run Regex CPF + ${measure_stab}= Will Parallel Test Be Run Regex STB VAR @{temp_list}= @{EMPTY} VAR @{freq_list}= @{EMPTY} VAR @{stab_list}= @{EMPTY} # TODO temporary debug values - VAR ${FREQUENCY_TEST_MEASURE_INTERVAL}= 1 - VAR ${TEMPERATURE_TEST_MEASURE_INTERVAL}= 1 - VAR ${TEMPERATURE_TEST_DURATION}= 5 - VAR ${FREQUENCY_TEST_DURATION}= 5 - VAR ${STABILITY_TEST_DURATION}= 0 + VAR ${frequency_test_measure_interval}= 1 + VAR ${temperature_test_measure_interval}= 1 + VAR ${stability_test_measure_interval}= 1 + VAR ${temperature_test_duration}= 5 + VAR ${frequency_test_duration}= 5 + VAR ${stability_test_duration}= 5 - ${next_temp_time}= Set Variable If ${gather_temps} ${TEMPERATURE_TEST_MEASURE_INTERVAL} 999999 - ${next_stab_time}= Set Variable If ${gather_temps} ${STABILITY_TEST_MEASURE_INTERVAL} 999999 - ${next_freq_time}= Set Variable If ${gather_temps} ${FREQUENCY_TEST_MEASURE_INTERVAL} 999999 + Set Variable If ${gather_temps} ${temperature_test_measure_interval} 999999 + Set Variable If ${measure_stab} ${stability_test_measure_interval} 999999 + Set Variable If ${gather_freqs} ${frequency_test_measure_interval} 999999 ${start}= DateTime.Get Current Date ${timer}= Evaluate 0 - VAR ${longest_duration}= max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}, ${STABILITY_TEST_DURATION}) + VAR ${longest_duration}= + ... max(${temperature_test_duration}, ${frequency_test_duration}, ${stability_test_duration}) WHILE ${timer} < ${longest_duration} ${now}= Get Current Date ${timer}= Subtract Date From Date ${now} ${start} - IF ${gather_temps} and ${TEMPERATURE_TEST_DURATION} >= ${timer} >= ${next_temp_time} + IF ${gather_temps} and ${temperature_test_duration} >= ${timer} >= ${next_temp_time} ${temperature}= Get CPU Temperature - ${next_temp_time}= Evaluate ${timer} + ${TEMPERATURE_TEST_MEASURE_INTERVAL} + ${next_temp_time}= Evaluate ${timer} + ${temperature_test_measure_interval} Append To List ${temp_list} ${temperature} Log To Console ${timer}s: Temperature: ${temperature} END - IF ${gather_freqs} and ${FREQUENCY_TEST_DURATION} >= ${timer} >= ${next_freq_time} + IF ${gather_freqs} and ${frequency_test_duration} >= ${timer} >= ${next_freq_time} ${freqs}= Get CPU Frequencies In Ubuntu - ${next_freq_time}= Evaluate ${timer} + ${FREQUENCY_TEST_MEASURE_INTERVAL} + ${next_freq_time}= Evaluate ${timer} + ${frequency_test_measure_interval} Append To List ${freq_list} ${freqs} Log To Console ${timer}s: Frequencies: ${freqs} END @@ -204,12 +122,12 @@ _GATHER_Background Measurements (no load) (Ubuntu) Sleep ${time_to_next_interval} END - Set Test Outputs ${TESTS_TO_RUN} CPT001.201 ${temp_list} - Set Test Outputs ${TESTS_TO_RUN} CPF005.201 ${freq_list} + Set Parallel Test Outputs CPT001.201 ${temp_list} + Set Parallel Test Outputs CPF005.201 ${freq_list} CPT001.201 CPU temperature without load (Ubuntu) - VAR ${test_id}= CPT001.201 - ${temps}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + VAR ${PARALLEL_TEST_ID}= CPT001.201 + ${temps}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} ${sum}= Evaluate 0 ${len}= Get Length ${temps} FOR ${temp} IN @{temps} @@ -225,8 +143,8 @@ CPT001.201 CPU temperature without load (Ubuntu) Should Be True ${avg} < ${MAX_CPU_TEMP} Average is higher than threshold CPF005.201 CPU runs on expected frequency (Ubuntu) - VAR ${test_id}= CPF005.201 - ${freqs}= Get Test Outputs ${TESTS_TO_RUN} ${test_id} + VAR ${PARALLEL_TEST_ID}= CPF005.201 + ${freqs}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} ${cpu_max_frequency_tol}= Evaluate ${CPU_MAX_FREQUENCY} * 1.125 ${cpu_min_frequency_tol}= Evaluate ${CPU_MIN_FREQUENCY} * 0.875 FOR ${freqs_cores} IN @{freqs} @@ -236,74 +154,31 @@ CPF005.201 CPU runs on expected frequency (Ubuntu) END END -*** Keywords *** -Wait Until Time - [Arguments] ${target} - ${now}= Get Current Date result_format=%Y-%m-%d %H:%M:%S.%f - ${diff}= Subtract Date From Date ${target} ${now} result_format=number - ${sleep}= Evaluate max(min(${diff}, 0.1), 0.01) - Sleep ${sleep} - -Add Test Case Skip Condition - [Arguments] ${tests_dict} ${test_id} ${condition} ${skip_reason} - ${current_details}= Get From Dictionary ${tests_dict} ${test_id} default=${FALSE} - IF not ${current_details} - VAR &{support_details}= test=${test_id} run=${TRUE} reason=supported - ELSE - VAR ${support_details}= ${current_details} - END - ${run}= Get From Dictionary ${support_details} run - IF ${run} and (${condition}) - Set To Dictionary ${support_details} run=${FALSE} reason=${skip_reason} - END - Set To Dictionary ${tests_dict} ${test_id}=${support_details} - -Skip If Not Supported - [Arguments] ${test_id} ${tests_dict} - ${will_be_run}= Will Test Be Run ${tests_dict} ${test_id} - Skip If not $will_be_run - -Will Test Be Run - [Arguments] ${tests_dict} ${test_id} - ${test}= Get From Dictionary ${tests_dict} ${test_id} ${None} - IF $test is ${None} - Log Test ${test_id} not found - RETURN ${FALSE} - END - ${value}= Get From Dictionary ${test} run ${FALSE} - RETURN ${value} - - -Get Test Value - [Arguments] ${tests_dict} ${test} ${key} - ${test}= Get From Dictionary ${tests_dict} ${test} - ${value}= Get From Dictionary ${test} ${key} - RETURN ${value} - -Get Test Outputs - [Arguments] ${tests_dict} ${test} - ${outputs}= Get Test Value ${tests_dict} ${test} outputs - RETURN ${outputs} - -Set Test Value - [Arguments] ${tests_dict} ${test_id} ${key} ${value} - ${test}= Get From Dictionary ${tests_dict} ${test_id} - Set To Dictionary ${test} ${key}=${value} - Set To Dictionary ${tests_dict} ${test_id}=${test} - -Set Test Outputs - [Arguments] ${tests_dict} ${test} ${value} - Set Test Value ${tests_dict} ${test} outputs ${value} *** Keywords *** -Will Test Be Run Regex - [Arguments] ${dict} ${pattern} - FOR ${key} IN @{dict.keys()} - ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re - IF $match is not ${None} - ${id}= Evaluate $match.string - ${run}= Will Test Be Run ${TESTS_TO_RUN} ${id} - IF ${run} RETURN ${TRUE} - END - END - RETURN ${FALSE} \ No newline at end of file +Prepare Parallel Test Suite + # Preparing parallel test cases + VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST + Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros + + VAR ${PARALLEL_TEST_ID}= CPF005.201 scope=TEST + Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros + Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + + VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros + Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + + Log To Console Tests to run: + ${tests}= Get Parallel Tests To Run + Log To Console ${tests} + + ${prepare_sensors}= Will Parallel Test Be Run Regex CP[TF] + IF ${prepare_sensors} Prepare Sensors + ${check_psu}= Will Parallel Test Be Run Regex CP[TF] + IF ${check_psu} Check Power Supply diff --git a/lib/parallel-testing.robot b/lib/parallel-testing.robot new file mode 100644 index 0000000000..91e7521f57 --- /dev/null +++ b/lib/parallel-testing.robot @@ -0,0 +1,114 @@ +*** Settings *** +Library DateTime +Library Collections + +*** Variables *** +&{PARALLEL_TESTS}= &{EMPTY} +${PARALLEL_TEST_ID}= ${EMPTY} + +*** Keywords *** +Init Parallel Testing + [Documentation] Initializes variables used by the parallel testing library. + ... Needs to be called before any other library keyword is called. + VAR &{PARALLEL_TESTS}= &{EMPTY} scope=GLOBAL + VAR ${PARALLEL_TEST_ID}= ${EMPTY} scope=GLOBAL + +Wait Until Time + [Documentation] Sleeps until given DateTime is reached. + ... Not accurate, will most likely wait a bit more, but never less than required. + [Arguments] ${target} + ${now}= Get Current Date result_format=%Y-%m-%d %H:%M:%S.%f + ${diff}= Subtract Date From Date ${target} ${now} result_format=number + ${sleep}= Evaluate max(min(${diff}, 0.1), 0.01) + Sleep ${sleep} + +Add Parallel Test Skip Condition + [Documentation] Creates a parallel test context if does not exist. + ... Verifies a skip condition immediately and saves the skip state and + ... a skip reason if the skip condition is true. + [Arguments] ${condition} ${skip_reason} + # [Globals] ${TESTS_TO_RUN} ${TEST_ID} + ${current_details}= Get From Dictionary ${PARALLEL_TESTS} ${PARALLEL_TEST_ID} default=${FALSE} + IF not ${current_details} + VAR &{support_details}= test=${PARALLEL_TEST_ID} run=${TRUE} skip_reason=supported + ELSE + VAR ${support_details}= ${current_details} + END + ${run}= Get From Dictionary ${support_details} run + IF ${run} and (${condition}) + Set To Dictionary ${support_details} run=${FALSE} skip_reason=${skip_reason} + END + Set To Dictionary ${PARALLEL_TESTS} ${PARALLEL_TEST_ID}=${support_details} + +Skip If Parallel Test Not Supported + [Documentation] Skips execution if the parallel test context shows that + ... the test should be skipped. Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${test_id} + ${will_be_run}= Will Parallel Test Be Run ${test_id} + ${reason}= Get Parallel Test Value ${test_id} skip_reason + Skip If not $will_be_run ${reason} + +Will Parallel Test Be Run + [Documentation] Returns TRUE/FALSE if the given parallel test case will + ... be run. Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${test_id} + ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test_id} ${None} + IF $test is ${None} + Log Test ${test_id} not found + RETURN ${FALSE} + END + ${value}= Get From Dictionary ${test} run ${FALSE} + RETURN ${value} + +Get Parallel Test Value + [Documentation] Returns a value for a given key in a parallel test context. + [Arguments] ${test} ${key} + ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test} + ${value}= Get From Dictionary ${test} ${key} + RETURN ${value} + +Get Parallel Test Outputs + [Documentation] Returns the value of `outputs` key of a parallel test context. + [Arguments] ${test_id} + ${outputs}= Get Parallel Test Value ${test_id} outputs + RETURN ${outputs} + +Set Parallel Test Value + [Documentation] Sets the value of a given key for a parallel test context. + [Arguments] ${test_id} ${key} ${value} + ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test_id} + Set To Dictionary ${test} ${key}=${value} + Set To Dictionary ${PARALLEL_TESTS} ${test_id}=${test} + +Set Parallel Test Outputs + [Documentation] Sets the value of `output` key for a parallel test context. + [Arguments] ${test} ${value} + Set Parallel Test Value ${test} outputs ${value} + +Get Parallel Tests To Run + [Documentation] Returns a list of all the parallel test cases that are + ... to be run according to their skip conditions. + ... Configured using `Add Parallel Test Skip Condition`. + ${tests}= Create List + FOR ${key} IN @{PARALLEL_TESTS.keys()} + VAR ${run}= ${PARALLEL_TESTS["${key}"]["run"]} + IF ${run} Append To List ${tests} ${key} + END + RETURN ${tests} + +Will Parallel Test Be Run Regex + [Documentation] Returns TRUE/FALSE if any parallel test case, the name of + ... which matches a given regular expression, will be run. + ... Helpful for determining whether a preparation step should be run by + ... checking if any test from a group of tests will be run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${pattern} + FOR ${key} IN @{PARALLEL_TESTS.keys()} + ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re + IF $match is not ${None} + ${id}= Evaluate $match.string + ${run}= Will Parallel Test Be Run ${id} + IF ${run} RETURN ${TRUE} + END + END + RETURN ${FALSE} \ No newline at end of file From 2b437da35d003ff9fd1913b31a814e43f4f00f7d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Tue, 21 Oct 2025 16:34:57 +0200 Subject: [PATCH 06/13] measurements.robot: Separate repeated code into kwds MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 183 ++++++++++-------- 1 file changed, 103 insertions(+), 80 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 645a4f5ad0..91262a0f74 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -4,7 +4,6 @@ Library DateTime Library String Library Telnet timeout=20 seconds connection_timeout=120 seconds Library SSHLibrary timeout=90 seconds - Resource ../variables.robot Resource ../keywords.robot Resource ../lib/performance/cpu.robot @@ -12,11 +11,11 @@ Resource ../lib/sensors/sensors.robot Resource ../lib/parallel-testing.robot Suite Setup Run Keywords -... Prepare Test Suite -... AND Run Keyword If "CPT" in " ".join($TEST_CASES) or "CPF" in " ".join($TEST_CASES) -... Check Power Supply -... AND Init Parallel Testing -... AND Prepare Parallel Test Suite +... Prepare Test Suite +... AND Run Keyword If "CPT" in " ".join($TEST_CASES) or "CPF" in " ".join($TEST_CASES) +... Check Power Supply +... AND Init Parallel Testing +... AND Prepare Parallel Test Suite *** Test Cases *** @@ -25,7 +24,8 @@ Suite Setup Run Keywords ############################################ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # immediately skip if no tests want these measurements - ${will_any_be_run}= Will Parallel Test Be Run Regex CPF001.201 + ${will_any_be_run}= Will Parallel Test Be Run Regex + ... (CPF001).201 Skip If not ${will_any_be_run} Power On @@ -34,35 +34,30 @@ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) Switch To Root User # CPF001.201 steps - VAR ${PARALLEL_TEST_ID}= CPF001.201 - ${check_frequency}= Will Parallel Test Be Run ${PARALLEL_TEST_ID} + VAR ${parallel_test_id}= CPF001.201 + ${check_frequency}= Will Parallel Test Be Run ${parallel_test_id} IF ${check_frequency} Sleep 10s @{frequencies}= Get CPU Frequencies In Ubuntu - Set Parallel Test Outputs ${PARALLEL_TEST_ID} ${frequencies} + Set Parallel Test Outputs ${parallel_test_id} ${frequencies} END CPF001.201 CPU not stuck on initial frequency (Ubuntu) [Documentation] This test aims to verify whether the mounted CPU does not ... stuck on the initial frequency after booting into the OS. ... Previous IDs: CPF001.001 - VAR ${PARALLEL_TEST_ID}= CPF001.201 - Skip If Parallel Test Not Supported ${PARALLEL_TEST_ID} - @{frequencies}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} - ${first_frequency}= Get From List ${frequencies} 0 - FOR ${frequency} IN @{frequencies} - IF ${frequency} != ${INITIAL_CPU_FREQUENCY} - Pass Execution CPU does not stuck on initial frequency - END - END - Fail CPU stuck on initial frequency: ${INITIAL_CPU_FREQUENCY} + VAR ${parallel_test_id}= CPF001.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${outs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Frequencies Not Stuck ${outs} ############################################################################# # Tests that gather measurements on Ubuntu, no load, n/a power source # ############################################################################# + _PARALLEL_Background Measurements (no load) (Ubuntu) ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF005)|(CPT001)|(STB001).201 + ... (CPF005)|(CPT001).201 Skip If not ${will_any_be_run} Power On @@ -70,92 +65,81 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) Login To Linux Switch To Root User - # CPF001.201 steps - ${gather_temps}= Will Parallel Test Be Run Regex CPT - ${gather_freqs}= Will Parallel Test Be Run Regex CPF - ${measure_stab}= Will Parallel Test Be Run Regex STB - - VAR @{temp_list}= @{EMPTY} - VAR @{freq_list}= @{EMPTY} - VAR @{stab_list}= @{EMPTY} - # TODO temporary debug values VAR ${frequency_test_measure_interval}= 1 VAR ${temperature_test_measure_interval}= 1 - VAR ${stability_test_measure_interval}= 1 VAR ${temperature_test_duration}= 5 VAR ${frequency_test_duration}= 5 - VAR ${stability_test_duration}= 5 - Set Variable If ${gather_temps} ${temperature_test_measure_interval} 999999 - Set Variable If ${measure_stab} ${stability_test_measure_interval} 999999 - Set Variable If ${gather_freqs} ${frequency_test_measure_interval} 999999 + ${gather_temps}= Will Parallel Test Be Run Regex CPT + ${gather_freqs}= Will Parallel Test Be Run Regex CPF + ${gather_temps}= Set Variable If ${gather_temps} CPT001.201 ${None} + ${gather_freqs}= Set Variable If ${gather_freqs} CPF005.201 ${None} + Background Measurements + ... id_temp=${gather_temps} id_freq=${gather_freqs} + +CPT001.201 CPU temperature without load (Ubuntu) + VAR ${parallel_test_id}= CPT001.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${temps}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Temps ${temps} + + +CPF005.201 CPU runs on expected frequency (Ubuntu) + VAR ${parallel_test_id}= CPF005.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${freqs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Freqs ${freqs} + + +*** Keywords *** +Background Measurements + [Arguments] ${id_temp}=${None} ${id_freq}=${None} + # Initialization + VAR @{temp_list}= @{EMPTY} + VAR @{freq_list}= @{EMPTY} + IF ${id_temp} is not ${None} + VAR ${next_temp_time}= 0 + ELSE + VAR ${next_temp_time}= 999999 + END + IF ${id_freq} is not ${None} + VAR ${next_freq_time}= 0 + ELSE + VAR ${next_freq_time}= 999999 + END + VAR ${longest_duration}= + ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}) ${start}= DateTime.Get Current Date ${timer}= Evaluate 0 - VAR ${longest_duration}= - ... max(${temperature_test_duration}, ${frequency_test_duration}, ${stability_test_duration}) + # measurement loop WHILE ${timer} < ${longest_duration} ${now}= Get Current Date ${timer}= Subtract Date From Date ${now} ${start} - IF ${gather_temps} and ${temperature_test_duration} >= ${timer} >= ${next_temp_time} + IF ${TEMPERATURE_TEST_DURATION} >= ${timer} >= ${next_temp_time} ${temperature}= Get CPU Temperature - ${next_temp_time}= Evaluate ${timer} + ${temperature_test_measure_interval} + ${next_temp_time}= Evaluate ${timer} + ${TEMPERATURE_TEST_MEASURE_INTERVAL} Append To List ${temp_list} ${temperature} Log To Console ${timer}s: Temperature: ${temperature} END - IF ${gather_freqs} and ${frequency_test_duration} >= ${timer} >= ${next_freq_time} + IF ${FREQUENCY_TEST_DURATION} >= ${timer} >= ${next_freq_time} ${freqs}= Get CPU Frequencies In Ubuntu - ${next_freq_time}= Evaluate ${timer} + ${frequency_test_measure_interval} + ${next_freq_time}= Evaluate ${timer} + ${FREQUENCY_TEST_MEASURE_INTERVAL} Append To List ${freq_list} ${freqs} Log To Console ${timer}s: Frequencies: ${freqs} END - # ... Log To Console Stability check at ${elapsed}s - - # Run Keyword If ${elapsed} % ${FREQUENCY_TEST_MEASURE_INTERVAL} < 0.1 and ${elapsed} <= ${FREQUENCY_TEST_DURATION} - # ... Log To Console Frequency check at ${elapsed}s - ${time_to_next_interval}= Evaluate min(${next_temp_time}, ${next_stab_time}, ${next_freq_time}) + ${time_to_next_interval}= Evaluate min(${next_temp_time}, ${next_freq_time}) Sleep ${time_to_next_interval} END - Set Parallel Test Outputs CPT001.201 ${temp_list} - Set Parallel Test Outputs CPF005.201 ${freq_list} + Set Parallel Test Outputs ${id_temp} ${temp_list} + Set Parallel Test Outputs ${id_freq} ${freq_list} -CPT001.201 CPU temperature without load (Ubuntu) - VAR ${PARALLEL_TEST_ID}= CPT001.201 - ${temps}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} - ${sum}= Evaluate 0 - ${len}= Get Length ${temps} - FOR ${temp} IN @{temps} - ${sum}= Evaluate ${sum} + ${temp} - END - ${avg}= Evaluate ${sum} / ${len} - ${min}= Evaluate min($temps) - ${max}= Evaluate max($temps) - Log To Console Average temp: ${avg} - Log To Console Min temp: ${min} - Log To Console Max temp: ${max} - Log To Console Test threshold of CPU temp: ${MAX_CPU_TEMP}°C - Should Be True ${avg} < ${MAX_CPU_TEMP} Average is higher than threshold - -CPF005.201 CPU runs on expected frequency (Ubuntu) - VAR ${PARALLEL_TEST_ID}= CPF005.201 - ${freqs}= Get Parallel Test Outputs ${PARALLEL_TEST_ID} - ${cpu_max_frequency_tol}= Evaluate ${CPU_MAX_FREQUENCY} * 1.125 - ${cpu_min_frequency_tol}= Evaluate ${CPU_MIN_FREQUENCY} * 0.875 - FOR ${freqs_cores} IN @{freqs} - FOR ${core} IN @{freqs_cores} - ${in_range}= Evaluate ${cpu_min_frequency_tol} <= ${core} <= ${cpu_max_frequency_tol} - Should Be True ${in_range} Encountered invalid frequency: ${in_range} MHz - END - END - - -*** Keywords *** Prepare Parallel Test Suite # Preparing parallel test cases VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST @@ -171,7 +155,9 @@ Prepare Parallel Test Suite VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported - Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros + Add Parallel Test Skip Condition + ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop Log To Console Tests to run: @@ -182,3 +168,40 @@ Prepare Parallel Test Suite IF ${prepare_sensors} Prepare Sensors ${check_psu}= Will Parallel Test Be Run Regex CP[TF] IF ${check_psu} Check Power Supply + +Check CPU Frequencies Not Stuck + [Arguments] ${frequencies} + ${first_frequency}= Get From List ${frequencies} 0 + FOR ${frequency} IN @{frequencies} + IF ${frequency} != ${INITIAL_CPU_FREQUENCY} + Pass Execution CPU does not stuck on initial frequency + END + END + Fail CPU stuck on initial frequency: ${INITIAL_CPU_FREQUENCY} + +Check CPU Temps + [Arguments] ${temps} + ${sum}= Evaluate 0 + ${len}= Get Length ${temps} + FOR ${temp} IN @{temps} + ${sum}= Evaluate ${sum} + ${temp} + END + ${avg}= Evaluate ${sum} / ${len} + ${min}= Evaluate min($temps) + ${max}= Evaluate max($temps) + Log To Console Average temp: ${avg} + Log To Console Min temp: ${min} + Log To Console Max temp: ${max} + Log To Console Test threshold of CPU temp: ${MAX_CPU_TEMP}°C + Should Be True ${avg} < ${MAX_CPU_TEMP} Average is higher than threshold + +Check CPU Freqs + [Arguments] ${freqs} + ${cpu_max_frequency_tol}= Evaluate ${CPU_MAX_FREQUENCY} * 1.125 + ${cpu_min_frequency_tol}= Evaluate ${CPU_MIN_FREQUENCY} * 0.875 + FOR ${freqs_cores} IN @{freqs} + FOR ${core} IN @{freqs_cores} + ${in_range}= Evaluate ${cpu_min_frequency_tol} <= ${core} <= ${cpu_max_frequency_tol} + Should Be True ${in_range} Encountered invalid frequency: ${in_range} MHz + END + END \ No newline at end of file From 7229bdd1e39a073733c04aeed7b864bd151a2203 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Tue, 21 Oct 2025 16:41:25 +0200 Subject: [PATCH 07/13] measurements.robot: Add Ubuntu tests under load MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 84 +++++++++++++++++-- 1 file changed, 79 insertions(+), 5 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 91262a0f74..94fbbd332a 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -73,10 +73,19 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) ${gather_temps}= Will Parallel Test Be Run Regex CPT ${gather_freqs}= Will Parallel Test Be Run Regex CPF - ${gather_temps}= Set Variable If ${gather_temps} CPT001.201 ${None} - ${gather_freqs}= Set Variable If ${gather_freqs} CPF005.201 ${None} + IF ${gather_temps} + VAR ${gather_temps}= CPT001.201 + ELSE + VAR ${gather_temps}= ${None} + END + IF ${gather_freqs} + VAR ${gather_freqs}= CPF005.201 + ELSE + VAR ${gather_freqs}= ${None} + END + Background Measurements - ... id_temp=${gather_temps} id_freq=${gather_freqs} + ... id_temp=${gather_temps} id_freq=${gather_freqs} CPT001.201 CPU temperature without load (Ubuntu) VAR ${parallel_test_id}= CPT001.201 @@ -84,13 +93,65 @@ CPT001.201 CPU temperature without load (Ubuntu) ${temps}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Temps ${temps} - CPF005.201 CPU runs on expected frequency (Ubuntu) VAR ${parallel_test_id}= CPF005.201 Skip If Parallel Test Not Supported ${parallel_test_id} ${freqs}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Freqs ${freqs} +############################################################################# +# Tests that gather measurements on Ubuntu, load, n/a power source # +############################################################################# + +_PARALLEL_Background Measurements (load) (Ubuntu) + ${will_any_be_run}= Will Parallel Test Be Run Regex + ... (CPF005)|(CPT001).201 + Skip If not ${will_any_be_run} + + Power On + Boot System Or From Connected Disk ${ENV_ID_UBUNTU} + Login To Linux + Switch To Root User + + # TODO temporary debug values + VAR ${frequency_test_measure_interval}= 1 + VAR ${temperature_test_measure_interval}= 1 + VAR ${temperature_test_duration}= 5 + VAR ${frequency_test_duration}= 5 + + ${gather_temps}= Will Parallel Test Be Run Regex CPT + ${gather_freqs}= Will Parallel Test Be Run Regex CPF + IF ${gather_temps} + VAR ${gather_temps}= CPT005.201 + ELSE + VAR ${gather_temps}= ${None} + END + IF ${gather_freqs} + VAR ${gather_freqs}= CPF009.201 + ELSE + VAR ${gather_freqs}= ${None} + END + # Start CPU Stress + ${stress_duration}= Evaluate + ... max(${temperature_test_duration}, ${frequency_test_duration}) + Stress Test ${stress_duration}s + Background Measurements + ... id_temp=${gather_temps} id_freq=${gather_freqs} + # Make sure to stop any CPU stress after we end + Execute Command In Terminal pkill stress-ng + +CPT005.201 CPU temperature without load (Ubuntu) + VAR ${parallel_test_id}= CPT001.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${temps}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Temps ${temps} + +CPF009.201 CPU runs on expected frequency (Ubuntu) + VAR ${parallel_test_id}= CPF005.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${freqs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Freqs ${freqs} + *** Keywords *** Background Measurements @@ -152,14 +213,27 @@ Prepare Parallel Test Suite Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + VAR ${PARALLEL_TEST_ID}= CPF009.201 scope=TEST + Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros + Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + #CPT VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported Add Parallel Test Skip Condition ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} ... Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + VAR ${PARALLEL_TEST_ID}= CPT005.201 scope=TEST + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + # Log to console to inform tester about the scope Log To Console Tests to run: ${tests}= Get Parallel Tests To Run Log To Console ${tests} @@ -204,4 +278,4 @@ Check CPU Freqs ${in_range}= Evaluate ${cpu_min_frequency_tol} <= ${core} <= ${cpu_max_frequency_tol} Should Be True ${in_range} Encountered invalid frequency: ${in_range} MHz END - END \ No newline at end of file + END From dc656e7978f74190d086e6ebee62c2d1f3573bd7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Tue, 21 Oct 2025 16:45:20 +0200 Subject: [PATCH 08/13] measurements.robot: Fix temporary debug perf tests duration vars MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 22 ++++++++----------- 1 file changed, 9 insertions(+), 13 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 94fbbd332a..3a9588bf0d 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -17,6 +17,14 @@ Suite Setup Run Keywords ... AND Init Parallel Testing ... AND Prepare Parallel Test Suite +*** Variables *** +# TODO: remove, temporary debug values +${FREQUENCY_TEST_MEASURE_INTERVAL}= 1 +${TEMPERATURE_TEST_MEASURE_INTERVAL}= 1 +${STABILITY_TEST_MEASURE_INTERVAL}= 1 +${TEMPERATURE_TEST_DURATION}= 5 +${FREQUENCY_TEST_DURATION}= 5 +${STABILITY_TEST_DURATION}= 5 *** Test Cases *** ############################################ @@ -65,12 +73,6 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) Login To Linux Switch To Root User - # TODO temporary debug values - VAR ${frequency_test_measure_interval}= 1 - VAR ${temperature_test_measure_interval}= 1 - VAR ${temperature_test_duration}= 5 - VAR ${frequency_test_duration}= 5 - ${gather_temps}= Will Parallel Test Be Run Regex CPT ${gather_freqs}= Will Parallel Test Be Run Regex CPF IF ${gather_temps} @@ -113,12 +115,6 @@ _PARALLEL_Background Measurements (load) (Ubuntu) Login To Linux Switch To Root User - # TODO temporary debug values - VAR ${frequency_test_measure_interval}= 1 - VAR ${temperature_test_measure_interval}= 1 - VAR ${temperature_test_duration}= 5 - VAR ${frequency_test_duration}= 5 - ${gather_temps}= Will Parallel Test Be Run Regex CPT ${gather_freqs}= Will Parallel Test Be Run Regex CPF IF ${gather_temps} @@ -133,7 +129,7 @@ _PARALLEL_Background Measurements (load) (Ubuntu) END # Start CPU Stress ${stress_duration}= Evaluate - ... max(${temperature_test_duration}, ${frequency_test_duration}) + ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}) Stress Test ${stress_duration}s Background Measurements ... id_temp=${gather_temps} id_freq=${gather_freqs} From 105f8376a3d7f080f56d451acee694cefb7bf846 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Tue, 21 Oct 2025 18:04:42 +0200 Subject: [PATCH 09/13] measurements.robot: Add STB test cases MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 116 ++++++++++++++++-- 1 file changed, 108 insertions(+), 8 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 3a9588bf0d..9af118f700 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -33,7 +33,7 @@ ${STABILITY_TEST_DURATION}= 5 _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # immediately skip if no tests want these measurements ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF001).201 + ... (CPF001)|(STB002).201 Skip If not ${will_any_be_run} Power On @@ -50,6 +50,14 @@ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) Set Parallel Test Outputs ${parallel_test_id} ${frequencies} END + # STB002.201 steps + VAR ${parallel_test_id}= STB002.201 + ${check_logs}= Will Parallel Test Be Run ${parallel_test_id} + IF ${check_logs} + ${dmesg_err_txt}= Execute Linux Command dmesg -t -l err,crit,alert,emerg + Set Parallel Test Outputs ${parallel_test_id} ${dmesg_err_txt} + END + CPF001.201 CPU not stuck on initial frequency (Ubuntu) [Documentation] This test aims to verify whether the mounted CPU does not ... stuck on the initial frequency after booting into the OS. @@ -59,13 +67,22 @@ CPF001.201 CPU not stuck on initial frequency (Ubuntu) ${outs}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Frequencies Not Stuck ${outs} +STB002.201 Verify if no unexpected boot errors appear in Linux logs + [Documentation] This test aims to verify that there are no unexpected + ... error ,essages in Linux kernel logs. + ... Previous IDs: STB002.001 + VAR ${parallel_test_id}= STB001.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${outs}= Get Parallel Test Outputs ${parallel_test_id} + Check Unexpected Boot Errors ${outs} + ############################################################################# # Tests that gather measurements on Ubuntu, no load, n/a power source # ############################################################################# _PARALLEL_Background Measurements (no load) (Ubuntu) ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF005)|(CPT001).201 + ... (CPF005)|(CPT001)|(STB001).201 Skip If not ${will_any_be_run} Power On @@ -75,6 +92,7 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) ${gather_temps}= Will Parallel Test Be Run Regex CPT ${gather_freqs}= Will Parallel Test Be Run Regex CPF + ${gather_stab}= Will Parallel Test Be Run Regex STB IF ${gather_temps} VAR ${gather_temps}= CPT001.201 ELSE @@ -85,6 +103,11 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) ELSE VAR ${gather_freqs}= ${None} END + IF ${gather_stab} + VAR ${gather_stab}= STB001.201 + ELSE + VAR ${gather_stab}= ${None} + END Background Measurements ... id_temp=${gather_temps} id_freq=${gather_freqs} @@ -101,13 +124,24 @@ CPF005.201 CPU runs on expected frequency (Ubuntu) ${freqs}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Freqs ${freqs} +STB001.201 Verify if no reboot occurs in the OS (Ubuntu) + [Documentation] This test aims to verify that the DUT booted to the + ... Operating System does not reset. The test is performed in multiple + ... iterations - after a defined time an attempt to read the output of + ... specific commands confirming the stability of work is repeated. + ... Previous IDs: STB001.002 + VAR ${parallel_test_id}= STB001.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${measurements}= Get Parallel Test Outputs ${parallel_test_id} + Check Platform Stability ${measurements} + ############################################################################# # Tests that gather measurements on Ubuntu, load, n/a power source # ############################################################################# _PARALLEL_Background Measurements (load) (Ubuntu) ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF005)|(CPT001).201 + ... (CPF005)|(CPT001)|(STB001).201 Skip If not ${will_any_be_run} Power On @@ -151,10 +185,11 @@ CPF009.201 CPU runs on expected frequency (Ubuntu) *** Keywords *** Background Measurements - [Arguments] ${id_temp}=${None} ${id_freq}=${None} + [Arguments] ${id_temp}=${None} ${id_freq}=${None} ${id_stab}=${None} # Initialization VAR @{temp_list}= @{EMPTY} VAR @{freq_list}= @{EMPTY} + VAR @{stab_list}= @{EMPTY} IF ${id_temp} is not ${None} VAR ${next_temp_time}= 0 ELSE @@ -165,9 +200,13 @@ Background Measurements ELSE VAR ${next_freq_time}= 999999 END - + IF ${id_stab} is not ${None} + VAR ${next_stab_time}= 0 + ELSE + VAR ${next_stab_time}= 999999 + END VAR ${longest_duration}= - ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}) + ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}, ${STABILITY_TEST_DURATION}) ${start}= DateTime.Get Current Date ${timer}= Evaluate 0 @@ -190,20 +229,41 @@ Background Measurements Log To Console ${timer}s: Frequencies: ${freqs} END - ${time_to_next_interval}= Evaluate min(${next_temp_time}, ${next_freq_time}) + IF ${FREQUENCY_TEST_DURATION} >= ${timer} >= ${next_freq_time} + ${network_status}= Execute Command In Terminal ip link | grep -E 'enp|eno' | grep -Eo 'UP|DOWN' + ${uptime_output}= Execute Command In Terminal cat /proc/uptime + ${uptime_list}= Split String ${uptime_output} ${SPACE} + ${current_uptime}= Convert To Number ${uptime_list}[0] + VAR &{stab_data}= network=${network_status} uptime=${current_uptime} + ${next_stab_time}= Evaluate ${timer} + ${FREQUENCY_TEST_MEASURE_INTERVAL} + Append To List ${freq_list} ${stab_data} + Log To Console ${timer}s: Stability: uptime ${current_uptime} + END + + ${time_to_next_interval}= Evaluate min(${next_temp_time}, ${next_stab_time}, ${next_freq_time}) Sleep ${time_to_next_interval} END Set Parallel Test Outputs ${id_temp} ${temp_list} Set Parallel Test Outputs ${id_freq} ${freq_list} + Set Parallel Test Outputs ${id_stab} ${stab_list} Prepare Parallel Test Suite # Preparing parallel test cases + # STB + VAR ${PARALLEL_TEST_ID}= STB001.201 scope=TEST + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB001.201 not supported + Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB001.201 not supported + VAR ${PARALLEL_TEST_ID}= STB002.201 scope=TEST + Add Parallel Test Skip Condition not ${PLATFORM_STABILITY_CHECKING} STB002.201 not supported + Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB002.201 not supported + Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB002.201 not supported + + #CPF VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros - VAR ${PARALLEL_TEST_ID}= CPF005.201 scope=TEST Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported @@ -275,3 +335,43 @@ Check CPU Freqs Should Be True ${in_range} Encountered invalid frequency: ${in_range} MHz END END + +Check Platform Stability + [Documentation] Check if a list of stability measurements shows + ... the platform is stable + [Arguments] ${measurements} + ${last_uptime}= Evaluate 0 + FOR ${measurement} IN @{measurements} + # no reboot since previous measurement + Should Be True float(${measurement["uptime"]}) > float(${last_uptime}) + # the network interface is up + Should Be Equal ${measurement["network"]} UP + END + +Check Unexpected Boot Errors + [Documentation] This keyword checks if any unexpected boot messages + ... appear in kernel logs. Messages with loglevel 3 (error) or lower + ... (more critical) are considered. + [Arguments] ${log} + VAR @{dmesg_err_allowlist}= @{EMPTY} + # Harmless error on Bluetooth modules + Append To List ${dmesg_err_allowlist} Bluetooth: hci0: Malformed MSFT vendor event: 0x02 + # Intel AX-series WiFi+BT adapters throw these when debug features are disabled + Append To List ${dmesg_err_allowlist} Bluetooth: hci0: No support for _PRR ACPI method + Append To List ${dmesg_err_allowlist} iwlwifi 0000:00:14.3: WRT: Invalid buffer destination + Append To List + ... ${dmesg_err_allowlist} + ... iwlwifi 0000:00:14.3: Not valid error log pointer 0x0027B0C0 for RT uCode + # GSC firmware loading via MEI fails when ME is disabled - not our bug + Append To List + ... ${dmesg_err_allowlist} + ... i915 0000:00:02.0: [drm] *ERROR* GT1: GSC proxy component didn't bind within the expected timeout + Append To List ${dmesg_err_allowlist} i915 0000:00:02.0: [drm] *ERROR* GT1: GSC proxy handler failed to init + # Not our bug + Append To List ${dmesg_err_allowlist} proc_thermal_pci 0000:00:04.0: error: proc_thermal_add, will continue + Append To List ${dmesg_err_allowlist} tmpfs: Unsupported parameter 'huge' + Append To List ${dmesg_err_allowlist} x86/mktme: No known encryption algorithm is supported: 0x4 + @{log}= Split To Lines ${log} + FOR ${error} IN @{log} + Should Contain ${dmesg_err_allowlist} ${error} + END \ No newline at end of file From f2ecfaf9cee5b378812f204567c180cac6b47578 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 22 Oct 2025 06:28:36 +0200 Subject: [PATCH 10/13] measurements.robot: Add docstrings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 59 ++++++++++++++----- lib/parallel-testing.robot | 12 ++-- 2 files changed, 50 insertions(+), 21 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index 9af118f700..db26fcd184 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -12,19 +12,20 @@ Resource ../lib/parallel-testing.robot Suite Setup Run Keywords ... Prepare Test Suite -... AND Run Keyword If "CPT" in " ".join($TEST_CASES) or "CPF" in " ".join($TEST_CASES) -... Check Power Supply +... AND Check Power Supply ... AND Init Parallel Testing ... AND Prepare Parallel Test Suite + *** Variables *** # TODO: remove, temporary debug values -${FREQUENCY_TEST_MEASURE_INTERVAL}= 1 -${TEMPERATURE_TEST_MEASURE_INTERVAL}= 1 -${STABILITY_TEST_MEASURE_INTERVAL}= 1 -${TEMPERATURE_TEST_DURATION}= 5 -${FREQUENCY_TEST_DURATION}= 5 -${STABILITY_TEST_DURATION}= 5 +${FREQUENCY_TEST_MEASURE_INTERVAL}= 1 +${TEMPERATURE_TEST_MEASURE_INTERVAL}= 1 +${STABILITY_TEST_MEASURE_INTERVAL}= 1 +${TEMPERATURE_TEST_DURATION}= 5 +${FREQUENCY_TEST_DURATION}= 5 +${STABILITY_TEST_DURATION}= 5 + *** Test Cases *** ############################################ @@ -34,7 +35,7 @@ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # immediately skip if no tests want these measurements ${will_any_be_run}= Will Parallel Test Be Run Regex ... (CPF001)|(STB002).201 - Skip If not ${will_any_be_run} + Skip If not ${will_any_be_run} No test depends on this step Power On Boot System Or From Connected Disk ${ENV_ID_UBUNTU} @@ -83,7 +84,7 @@ STB002.201 Verify if no unexpected boot errors appear in Linux logs _PARALLEL_Background Measurements (no load) (Ubuntu) ${will_any_be_run}= Will Parallel Test Be Run Regex ... (CPF005)|(CPT001)|(STB001).201 - Skip If not ${will_any_be_run} + Skip If not ${will_any_be_run} No test depends on this step Power On Boot System Or From Connected Disk ${ENV_ID_UBUNTU} @@ -113,12 +114,19 @@ _PARALLEL_Background Measurements (no load) (Ubuntu) ... id_temp=${gather_temps} id_freq=${gather_freqs} CPT001.201 CPU temperature without load (Ubuntu) + [Documentation] This test aims to verify whether the temperature of CPU + ... cores after system booting is not higher than the maximum + ... allowed temperature. + ... Previous IDs: CPT001.001 VAR ${parallel_test_id}= CPT001.201 Skip If Parallel Test Not Supported ${parallel_test_id} ${temps}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Temps ${temps} CPF005.201 CPU runs on expected frequency (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU is + ... running on expected frequency. + ... Previous IDs: CPF002.001 VAR ${parallel_test_id}= CPF005.201 Skip If Parallel Test Not Supported ${parallel_test_id} ${freqs}= Get Parallel Test Outputs ${parallel_test_id} @@ -142,7 +150,7 @@ STB001.201 Verify if no reboot occurs in the OS (Ubuntu) _PARALLEL_Background Measurements (load) (Ubuntu) ${will_any_be_run}= Will Parallel Test Be Run Regex ... (CPF005)|(CPT001)|(STB001).201 - Skip If not ${will_any_be_run} + Skip If not ${will_any_be_run} No test depends on this step Power On Boot System Or From Connected Disk ${ENV_ID_UBUNTU} @@ -170,13 +178,20 @@ _PARALLEL_Background Measurements (load) (Ubuntu) # Make sure to stop any CPU stress after we end Execute Command In Terminal pkill stress-ng -CPT005.201 CPU temperature without load (Ubuntu) +CPT005.201 CPU temperature after stress test (Ubuntu) + [Documentation] This test aims to verify whether the temperature of the + ... CPU cores is not higher than the maximum allowed + ... temperature during stress test. + ... Previous IDs: CPT002.001 VAR ${parallel_test_id}= CPT001.201 Skip If Parallel Test Not Supported ${parallel_test_id} ${temps}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Temps ${temps} -CPF009.201 CPU runs on expected frequency (Ubuntu) +CPF009.201 CPU with load runs on expected frequency (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU is + ... running on expected frequency after stress test. + ... Previous IDs: CPF004.001 VAR ${parallel_test_id}= CPF005.201 Skip If Parallel Test Not Supported ${parallel_test_id} ${freqs}= Get Parallel Test Outputs ${parallel_test_id} @@ -185,6 +200,10 @@ CPF009.201 CPU runs on expected frequency (Ubuntu) *** Keywords *** Background Measurements + [Documentation] Keyword for gathering CPU temps, freqs and stability info + ... in parallel. Set '${id_*}' vars to a test case ID to save the results + ... for this type of measurements under a chosen ID. Set to none to skip + ... measurements of the given type. [Arguments] ${id_temp}=${None} ${id_freq}=${None} ${id_stab}=${None} # Initialization VAR @{temp_list}= @{EMPTY} @@ -249,6 +268,7 @@ Background Measurements Set Parallel Test Outputs ${id_stab} ${stab_list} Prepare Parallel Test Suite + [Documentation] Setup all parallel tests skip conditions and run preparation steps # Preparing parallel test cases # STB VAR ${PARALLEL_TEST_ID}= STB001.201 scope=TEST @@ -259,7 +279,7 @@ Prepare Parallel Test Suite Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB002.201 not supported Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB002.201 not supported - #CPF + # CPF VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported @@ -275,7 +295,7 @@ Prepare Parallel Test Suite Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop - #CPT + # CPT VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported Add Parallel Test Skip Condition @@ -294,12 +314,15 @@ Prepare Parallel Test Suite ${tests}= Get Parallel Tests To Run Log To Console ${tests} + # Preparation steps ${prepare_sensors}= Will Parallel Test Be Run Regex CP[TF] IF ${prepare_sensors} Prepare Sensors ${check_psu}= Will Parallel Test Be Run Regex CP[TF] IF ${check_psu} Check Power Supply Check CPU Frequencies Not Stuck + [Documentation] Check if a list of CPU frequencies shows them being stuck + ... to defaults. [Arguments] ${frequencies} ${first_frequency}= Get From List ${frequencies} 0 FOR ${frequency} IN @{frequencies} @@ -310,6 +333,8 @@ Check CPU Frequencies Not Stuck Fail CPU stuck on initial frequency: ${INITIAL_CPU_FREQUENCY} Check CPU Temps + [Documentation] Check if a list of temperature measurements shows + ... acceptable temperature values [Arguments] ${temps} ${sum}= Evaluate 0 ${len}= Get Length ${temps} @@ -326,6 +351,8 @@ Check CPU Temps Should Be True ${avg} < ${MAX_CPU_TEMP} Average is higher than threshold Check CPU Freqs + [Documentation] Check if a list of frequency measurements shows + ... acceptable frequency values [Arguments] ${freqs} ${cpu_max_frequency_tol}= Evaluate ${CPU_MAX_FREQUENCY} * 1.125 ${cpu_min_frequency_tol}= Evaluate ${CPU_MIN_FREQUENCY} * 0.875 @@ -374,4 +401,4 @@ Check Unexpected Boot Errors @{log}= Split To Lines ${log} FOR ${error} IN @{log} Should Contain ${dmesg_err_allowlist} ${error} - END \ No newline at end of file + END diff --git a/lib/parallel-testing.robot b/lib/parallel-testing.robot index 91e7521f57..51295afb6a 100644 --- a/lib/parallel-testing.robot +++ b/lib/parallel-testing.robot @@ -1,11 +1,13 @@ *** Settings *** -Library DateTime -Library Collections +Library Collections +Library DateTime + *** Variables *** -&{PARALLEL_TESTS}= &{EMPTY} +&{PARALLEL_TESTS}= &{EMPTY} ${PARALLEL_TEST_ID}= ${EMPTY} + *** Keywords *** Init Parallel Testing [Documentation] Initializes variables used by the parallel testing library. @@ -89,7 +91,7 @@ Get Parallel Tests To Run [Documentation] Returns a list of all the parallel test cases that are ... to be run according to their skip conditions. ... Configured using `Add Parallel Test Skip Condition`. - ${tests}= Create List + VAR @{tests}= @{EMPTY} FOR ${key} IN @{PARALLEL_TESTS.keys()} VAR ${run}= ${PARALLEL_TESTS["${key}"]["run"]} IF ${run} Append To List ${tests} ${key} @@ -111,4 +113,4 @@ Will Parallel Test Be Run Regex IF ${run} RETURN ${TRUE} END END - RETURN ${FALSE} \ No newline at end of file + RETURN ${FALSE} From 6a47569d3612514eac3940fb78dbad2ad6f5a663 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 22 Oct 2025 07:36:30 +0200 Subject: [PATCH 11/13] measurements.robot: Add possibility of performing one test on multiple gather steps MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 92 +++++-------- lib/parallel-testing.robot | 122 ++++++++++++++---- 2 files changed, 128 insertions(+), 86 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index db26fcd184..e1671f9a9f 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -13,8 +13,12 @@ Resource ../lib/parallel-testing.robot Suite Setup Run Keywords ... Prepare Test Suite ... AND Check Power Supply +... AND Prepare Sensors ... AND Init Parallel Testing -... AND Prepare Parallel Test Suite +... AND Prepare CPT +... AND Prepare CPF +... AND Prepare STB +... AND Print Parallel Tests Summary *** Variables *** @@ -33,7 +37,7 @@ ${STABILITY_TEST_DURATION}= 5 ############################################ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # immediately skip if no tests want these measurements - ${will_any_be_run}= Will Parallel Test Be Run Regex + ${will_any_be_run}= Check Parallel Test Supported Regex ... (CPF001)|(STB002).201 Skip If not ${will_any_be_run} No test depends on this step @@ -44,7 +48,7 @@ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # CPF001.201 steps VAR ${parallel_test_id}= CPF001.201 - ${check_frequency}= Will Parallel Test Be Run ${parallel_test_id} + ${check_frequency}= Check Parallel Test Supported ${parallel_test_id} IF ${check_frequency} Sleep 10s @{frequencies}= Get CPU Frequencies In Ubuntu @@ -53,7 +57,7 @@ _PARALLEL_Background Measurements Immediate (no load) (Ubuntu) # STB002.201 steps VAR ${parallel_test_id}= STB002.201 - ${check_logs}= Will Parallel Test Be Run ${parallel_test_id} + ${check_logs}= Check Parallel Test Supported ${parallel_test_id} IF ${check_logs} ${dmesg_err_txt}= Execute Linux Command dmesg -t -l err,crit,alert,emerg Set Parallel Test Outputs ${parallel_test_id} ${dmesg_err_txt} @@ -82,36 +86,22 @@ STB002.201 Verify if no unexpected boot errors appear in Linux logs ############################################################################# _PARALLEL_Background Measurements (no load) (Ubuntu) - ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF005)|(CPT001)|(STB001).201 - Skip If not ${will_any_be_run} No test depends on this step + ${gather_temps}= Will Parallel Test Be Run CPT001.201 + ${gather_freqs}= Will Parallel Test Be Run CPF005.201 + ${gather_stab}= Will Parallel Test Be Run STB001.201 + Skip If not (${gather_temps} or ${gather_freqs} or ${gather_stab}) No test depends on this step Power On Boot System Or From Connected Disk ${ENV_ID_UBUNTU} Login To Linux Switch To Root User - ${gather_temps}= Will Parallel Test Be Run Regex CPT - ${gather_freqs}= Will Parallel Test Be Run Regex CPF - ${gather_stab}= Will Parallel Test Be Run Regex STB - IF ${gather_temps} - VAR ${gather_temps}= CPT001.201 - ELSE - VAR ${gather_temps}= ${None} - END - IF ${gather_freqs} - VAR ${gather_freqs}= CPF005.201 - ELSE - VAR ${gather_freqs}= ${None} - END - IF ${gather_stab} - VAR ${gather_stab}= STB001.201 - ELSE - VAR ${gather_stab}= ${None} - END + ${gather_temps}= Evaluate "CPT001.201" if ${gather_temps} else ${None} + ${gather_freqs}= Evaluate "CPF005.201" if ${gather_freqs} else ${None} + ${gather_stab}= Evaluate "STB001.201" if ${gather_stab} else ${None} Background Measurements - ... id_temp=${gather_temps} id_freq=${gather_freqs} + ... id_temp=${gather_temps} id_freq=${gather_freqs} id_stab=${gather_stab} CPT001.201 CPU temperature without load (Ubuntu) [Documentation] This test aims to verify whether the temperature of CPU @@ -148,33 +138,26 @@ STB001.201 Verify if no reboot occurs in the OS (Ubuntu) ############################################################################# _PARALLEL_Background Measurements (load) (Ubuntu) - ${will_any_be_run}= Will Parallel Test Be Run Regex - ... (CPF005)|(CPT001)|(STB001).201 - Skip If not ${will_any_be_run} No test depends on this step + ${gather_temps}= Will Parallel Test Be Run CPT005.201 + ${gather_freqs}= Will Parallel Test Be Run CPF009.201 + ${gather_stab}= Will Parallel Test Be Run STB001.201 + Skip If not (${gather_temps} or ${gather_freqs} or ${gather_stab}) No test depends on this step Power On Boot System Or From Connected Disk ${ENV_ID_UBUNTU} Login To Linux Switch To Root User - ${gather_temps}= Will Parallel Test Be Run Regex CPT - ${gather_freqs}= Will Parallel Test Be Run Regex CPF - IF ${gather_temps} - VAR ${gather_temps}= CPT005.201 - ELSE - VAR ${gather_temps}= ${None} - END - IF ${gather_freqs} - VAR ${gather_freqs}= CPF009.201 - ELSE - VAR ${gather_freqs}= ${None} - END + ${gather_temps}= Evaluate "CPT005.201" if ${gather_temps} else ${None} + ${gather_freqs}= Evaluate "CPF009.201" if ${gather_freqs} else ${None} + ${gather_stab}= Evaluate "STB001.201" if ${gather_stab} else ${None} + # Start CPU Stress ${stress_duration}= Evaluate - ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}) + ... max(${TEMPERATURE_TEST_DURATION}, ${FREQUENCY_TEST_DURATION}, ${STABILITY_TEST_DURATION}) Stress Test ${stress_duration}s Background Measurements - ... id_temp=${gather_temps} id_freq=${gather_freqs} + ... id_temp=${gather_temps} id_freq=${gather_freqs} id_stab=${gather_stab} # Make sure to stop any CPU stress after we end Execute Command In Terminal pkill stress-ng @@ -267,10 +250,8 @@ Background Measurements Set Parallel Test Outputs ${id_freq} ${freq_list} Set Parallel Test Outputs ${id_stab} ${stab_list} -Prepare Parallel Test Suite - [Documentation] Setup all parallel tests skip conditions and run preparation steps - # Preparing parallel test cases - # STB +Prepare STB + [Documentation] Setup STB parallel test contexts VAR ${PARALLEL_TEST_ID}= STB001.201 scope=TEST Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB001.201 not supported Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB001.201 not supported @@ -279,7 +260,8 @@ Prepare Parallel Test Suite Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB002.201 not supported Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB002.201 not supported - # CPF +Prepare CPF + [Documentation] Setup CPF parallel test contexts VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported @@ -295,7 +277,8 @@ Prepare Parallel Test Suite Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop - # CPT +Prepare CPT + [Documentation] Setup CPT parallel test contexts VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported Add Parallel Test Skip Condition @@ -309,17 +292,6 @@ Prepare Parallel Test Suite ... Ubuntu not in tested distros Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop - # Log to console to inform tester about the scope - Log To Console Tests to run: - ${tests}= Get Parallel Tests To Run - Log To Console ${tests} - - # Preparation steps - ${prepare_sensors}= Will Parallel Test Be Run Regex CP[TF] - IF ${prepare_sensors} Prepare Sensors - ${check_psu}= Will Parallel Test Be Run Regex CP[TF] - IF ${check_psu} Check Power Supply - Check CPU Frequencies Not Stuck [Documentation] Check if a list of CPU frequencies shows them being stuck ... to defaults. diff --git a/lib/parallel-testing.robot b/lib/parallel-testing.robot index 51295afb6a..74edf31e64 100644 --- a/lib/parallel-testing.robot +++ b/lib/parallel-testing.robot @@ -42,15 +42,7 @@ Add Parallel Test Skip Condition END Set To Dictionary ${PARALLEL_TESTS} ${PARALLEL_TEST_ID}=${support_details} -Skip If Parallel Test Not Supported - [Documentation] Skips execution if the parallel test context shows that - ... the test should be skipped. Configured using `Add Parallel Test Skip Condition`. - [Arguments] ${test_id} - ${will_be_run}= Will Parallel Test Be Run ${test_id} - ${reason}= Get Parallel Test Value ${test_id} skip_reason - Skip If not $will_be_run ${reason} - -Will Parallel Test Be Run +Check Parallel Test Supported [Documentation] Returns TRUE/FALSE if the given parallel test case will ... be run. Configured using `Add Parallel Test Skip Condition`. [Arguments] ${test_id} @@ -62,11 +54,92 @@ Will Parallel Test Be Run ${value}= Get From Dictionary ${test} run ${FALSE} RETURN ${value} +Check Parallel Test Supported Regex + [Documentation] Returns TRUE/FALSE if any parallel test case, the name of + ... which matches a given regular expression, will be run. + ... Helpful for determining whether a preparation step should be run by + ... checking if any test from a group of tests will be run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${pattern} + FOR ${key} IN @{PARALLEL_TESTS.keys()} + ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re + IF $match is not ${None} + ${id}= Evaluate $match.string + ${run}= Check Parallel Test Supported ${id} + IF ${run} RETURN ${TRUE} + END + END + +Skip If Parallel Test Not Supported + [Documentation] Skips execution if the parallel test context shows that + ... the test should be skipped. Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${test_id} + ${will_be_run}= Check Parallel Test Supported ${test_id} + ${reason}= Get Parallel Test Value ${test_id} skip_reason + Skip If not $will_be_run ${reason} + +Skip If Parallel Test Not Supported Regex + [Documentation] Skips execution if the parallel test context shows that + ... the test should be skipped. Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${pattern} ${reason}=${None} + ${will_be_run}= Check Parallel Test Supported Regex ${pattern} + Skip If not $will_be_run ${reason} + RETURN ${FALSE} + +Will Parallel Test Be Run + [Documentation] Returns TRUE/FALSE if the parallel test case is supported = + ... and was not run already. + ... Helpful for determining whether a preparation step should be run by + ... checking if any test from a group of tests will be run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${test_id} + ${run}= Check Parallel Test Supported ${test_id} + ${outputs}= Get Parallel Test Outputs ${test_id} + IF ${run} and $outputs != ${None} RETURN ${TRUE} + RETURN ${FALSE} + +Will Parallel Test Be Run Regex + [Documentation] Returns TRUE/FALSE if any parallel test case, the name of + ... which matches a given regular expression, is supported and was not + ... run already. + ... Helpful for determining whether a preparation step should be run by + ... checking if any test from a group of tests will be run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${pattern} + FOR ${key} IN @{PARALLEL_TESTS.keys()} + ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re + IF $match is not ${None} + ${id}= Evaluate $match.string + ${run}= Check Parallel Test Supported ${id} + ${outputs}= Get Parallel Test Outputs ${id} + IF ${run} and $outputs != ${None} RETURN ${TRUE} + END + END + RETURN ${FALSE} + +Skip If Parallel Test Wont Be Run + [Documentation] Skips execution if the parallel test context shows that + ... the test should be skipped or the test was already run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${test_id} ${reason}=${None} + ${will_be_run}= Will Parallel Test Be Run ${test_id} + Skip If not $will_be_run ${reason} + RETURN ${FALSE} + +Skip If Parallel Test Wont Be Run Regex + [Documentation] Skips execution if the parallel test context shows that + ... the test should be skipped or the test was already run. + ... Configured using `Add Parallel Test Skip Condition`. + [Arguments] ${pattern} ${reason}=${None} + ${will_be_run}= Will Parallel Test Be Run Regex ${pattern} + Skip If not $will_be_run ${reason} + RETURN ${FALSE} + Get Parallel Test Value [Documentation] Returns a value for a given key in a parallel test context. [Arguments] ${test} ${key} ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test} - ${value}= Get From Dictionary ${test} ${key} + ${value}= Get From Dictionary ${test} ${key} ${EMPTY} RETURN ${value} Get Parallel Test Outputs @@ -98,19 +171,16 @@ Get Parallel Tests To Run END RETURN ${tests} -Will Parallel Test Be Run Regex - [Documentation] Returns TRUE/FALSE if any parallel test case, the name of - ... which matches a given regular expression, will be run. - ... Helpful for determining whether a preparation step should be run by - ... checking if any test from a group of tests will be run. - ... Configured using `Add Parallel Test Skip Condition`. - [Arguments] ${pattern} - FOR ${key} IN @{PARALLEL_TESTS.keys()} - ${match}= Evaluate re.search(r"""${pattern}""", """${key}""") re - IF $match is not ${None} - ${id}= Evaluate $match.string - ${run}= Will Parallel Test Be Run ${id} - IF ${run} RETURN ${TRUE} - END - END - RETURN ${FALSE} +Copy Parallel Test Outputs + [Documentation] Copies the outputs of one test to another. + ... Useful when both can be verified using the same logs. + [Arguments] ${test_id_source} ${test_id_dest} + ${outs}= Get Parallel Test Outputs ${test_id_source} + Set Parallel Test Outputs ${test_id_dest} ${outs} + +Print Parallel Tests Summary + [Documentation] Prints the summary of Parallel tests to be run + # Log to console to inform tester about the scope + Log To Console Tests to run: + ${tests}= Get Parallel Tests To Run + Log To Console ${tests} \ No newline at end of file From 373bfd37bc6b4c6084c52e9f815cfd68d4baeee2 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 22 Oct 2025 08:16:19 +0200 Subject: [PATCH 12/13] measurements.robot: Add CPT CPF PSU variants MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- .../measurements.robot | 218 +++++++++++++++--- lib/parallel-testing.robot | 11 +- 2 files changed, 195 insertions(+), 34 deletions(-) diff --git a/dasharo-performance-parallelable/measurements.robot b/dasharo-performance-parallelable/measurements.robot index e1671f9a9f..e4a0b913b3 100644 --- a/dasharo-performance-parallelable/measurements.robot +++ b/dasharo-performance-parallelable/measurements.robot @@ -72,6 +72,33 @@ CPF001.201 CPU not stuck on initial frequency (Ubuntu) ${outs}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Frequencies Not Stuck ${outs} +CPF002.201 CPU not stuck on initial frequency (Battery) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU does not + ... stuck on the initial frequency after booting into the OS. + ... Previous IDs: CPF001.001 + VAR ${parallel_test_id}= CPF002.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${outs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Frequencies Not Stuck ${outs} + +CPF003.201 CPU not stuck on initial frequency (AC) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU does not + ... stuck on the initial frequency after booting into the OS. + ... Previous IDs: CPF001.001 + VAR ${parallel_test_id}= CPF003.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${outs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Frequencies Not Stuck ${outs} + +CPF004.201 CPU not stuck on initial frequency (USB-PD) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU does not + ... stuck on the initial frequency after booting into the OS. + ... Previous IDs: CPF001.001 + VAR ${parallel_test_id}= CPF004.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${outs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Frequencies Not Stuck ${outs} + STB002.201 Verify if no unexpected boot errors appear in Linux logs [Documentation] This test aims to verify that there are no unexpected ... error ,essages in Linux kernel logs. @@ -113,6 +140,36 @@ CPT001.201 CPU temperature without load (Ubuntu) ${temps}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Temps ${temps} +CPT002.201 CPU temperature without load (Battery) (Ubuntu) + [Documentation] This test aims to verify whether the temperature of CPU + ... cores after system booting is not higher than the maximum + ... allowed temperature. + ... Previous IDs: CPT001.001 + VAR ${parallel_test_id}= CPT002.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${temps}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Temps ${temps} + +CPT003.201 CPU temperature without load (AC) (Ubuntu) + [Documentation] This test aims to verify whether the temperature of CPU + ... cores after system booting is not higher than the maximum + ... allowed temperature. + ... Previous IDs: CPT001.001 + VAR ${parallel_test_id}= CPT003.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${temps}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Temps ${temps} + +CPT004.201 CPU temperature without load (USB-PD) (Ubuntu) + [Documentation] This test aims to verify whether the temperature of CPU + ... cores after system booting is not higher than the maximum + ... allowed temperature. + ... Previous IDs: CPT001.001 + VAR ${parallel_test_id}= CPT004.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${temps}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Temps ${temps} + CPF005.201 CPU runs on expected frequency (Ubuntu) [Documentation] This test aims to verify whether the mounted CPU is ... running on expected frequency. @@ -122,6 +179,33 @@ CPF005.201 CPU runs on expected frequency (Ubuntu) ${freqs}= Get Parallel Test Outputs ${parallel_test_id} Check CPU Freqs ${freqs} +CPF006.201 CPU runs on expected frequency (Battery) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU is + ... running on expected frequency. + ... Previous IDs: CPF002.001 + VAR ${parallel_test_id}= CPF006.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${freqs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Freqs ${freqs} + +CPF007.201 CPU runs on expected frequency (AC) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU is + ... running on expected frequency. + ... Previous IDs: CPF002.001 + VAR ${parallel_test_id}= CPF007.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${freqs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Freqs ${freqs} + +CPF008.201 CPU runs on expected frequency (USB-PD) (Ubuntu) + [Documentation] This test aims to verify whether the mounted CPU is + ... running on expected frequency. + ... Previous IDs: CPF002.001 + VAR ${parallel_test_id}= CPF008.201 + Skip If Parallel Test Not Supported ${parallel_test_id} + ${freqs}= Get Parallel Test Outputs ${parallel_test_id} + Check CPU Freqs ${freqs} + STB001.201 Verify if no reboot occurs in the OS (Ubuntu) [Documentation] This test aims to verify that the DUT booted to the ... Operating System does not reset. The test is performed in multiple @@ -252,45 +336,125 @@ Background Measurements Prepare STB [Documentation] Setup STB parallel test contexts - VAR ${PARALLEL_TEST_ID}= STB001.201 scope=TEST - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB001.201 not supported - Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB001.201 not supported - VAR ${PARALLEL_TEST_ID}= STB002.201 scope=TEST - Add Parallel Test Skip Condition not ${PLATFORM_STABILITY_CHECKING} STB002.201 not supported - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} STB002.201 not supported - Add Parallel Test Skip Condition '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} STB002.201 not supported + # Stability check + Add Parallel Test Skip Condition + ... STB001.201 + ... not ${PLATFORM_STABILITY_CHECKING} + ... Stability checking not supported + Add Parallel Test Skip Condition + ... STB001.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... Tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... STB001.201 + ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + # Linux dmesg check + Add Parallel Test Skip Condition + ... STB002.201 + ... not ${PLATFORM_STABILITY_CHECKING} + ... Stability checking not supported + Add Parallel Test Skip Condition + ... STB002.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... Tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... STB002.201 + ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros Prepare CPF [Documentation] Setup CPF parallel test contexts - VAR ${PARALLEL_TEST_ID}= CPF001.201 scope=TEST - Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported - Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros - VAR ${PARALLEL_TEST_ID}= CPF005.201 scope=TEST - Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported - Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros - Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop - VAR ${PARALLEL_TEST_ID}= CPF009.201 scope=TEST - Add Parallel Test Skip Condition not ${CPU_FREQUENCY_MEASURE} frequency measure not supported - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported - Add Parallel Test Skip Condition '201' not in ${TESTED_LINUX_DISTROS} Ubuntu not in tested distros - Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop + IF not ${LAPTOP_PLATFORM} + VAR ${CPF_STUCK_ID}= CPF001 scope=SUITE + VAR ${CPF_NO_LOAD_ID}= CPF005 scope=SUITE + VAR ${CPF_LOAD_ID}= CPF009 scope=SUITE + ELSE IF ${BATTERY_PRESENT} + VAR ${CPF_STUCK_ID}= CPF002 scope=SUITE + VAR ${CPF_NO_LOAD_ID}= CPF006 scope=SUITE + VAR ${CPF_LOAD_ID}= CPF010 scope=SUITE + ELSE IF ${AC_CONNECTED} + VAR ${CPF_STUCK_ID}= CPF003 scope=SUITE + VAR ${CPF_NO_LOAD_ID}= CPF007 scope=SUITE + VAR ${CPF_LOAD_ID}= CPF011 scope=SUITE + ELSE IF ${USB_PD_CONNECTED} + VAR ${CPF_STUCK_ID}= CPF004 scope=SUITE + VAR ${CPF_NO_LOAD_ID}= CPF008 scope=SUITE + VAR ${CPF_LOAD_ID}= CPF012 scope=SUITE + END + # Not Stuck Ubuntu + Add Parallel Test Skip Condition + ... ${CPF_STUCK_ID}.201 + ... not ${CPU_FREQUENCY_MEASURE} + ... frequency measure not supported + Add Parallel Test Skip Condition + ... ${CPF_STUCK_ID}.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... ${CPF_STUCK_ID}.201 + ... '201' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + # No load Ubuntu + Add Parallel Test Skip Condition + ... ${CPF_NO_LOAD_ID}.201 + ... not ${CPU_FREQUENCY_MEASURE} + ... frequency measure not supported + Add Parallel Test Skip Condition + ... ${CPF_NO_LOAD_ID}.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... ${CPF_NO_LOAD_ID}.201 + ... '201' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros + # Load Ubuntu + Add Parallel Test Skip Condition + ... ${CPF_LOAD_ID}.201 + ... not ${CPU_FREQUENCY_MEASURE} + ... frequency measure not supported + Add Parallel Test Skip Condition + ... ${CPF_LOAD_ID}.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Parallel Test Skip Condition + ... ${CPF_LOAD_ID}.201 + ... '201' not in ${TESTED_LINUX_DISTROS} + ... Ubuntu not in tested distros Prepare CPT [Documentation] Setup CPT parallel test contexts - VAR ${PARALLEL_TEST_ID}= CPT001.201 scope=TEST - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + IF not ${LAPTOP_PLATFORM} + VAR ${CPT_NO_LOAD_ID}= CPT001 scope=SUITE + VAR ${CPF_LOAD_ID}= CPT005 scope=SUITE + ELSE IF ${BATTERY_PRESENT} + VAR ${CPT_NO_LOAD_ID}= CPT002 scope=SUITE + VAR ${CPF_LOAD_ID}= CPT006 scope=SUITE + ELSE IF ${AC_CONNECTED} + VAR ${CPT_NO_LOAD_ID}= CPT003 scope=SUITE + VAR ${CPF_LOAD_ID}= CPT007 scope=SUITE + ELSE IF ${USB_PD_CONNECTED} + VAR ${CPT_NO_LOAD_ID}= CPT004 scope=SUITE + VAR ${CPF_LOAD_ID}= CPT008 scope=SUITE + END + + # No load Ubuntu + Add Parallel Test Skip Condition + ... ${CPT_NO_LOAD_ID}.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported Add Parallel Test Skip Condition + ... ${CPT_NO_LOAD_ID}.201 ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} ... Ubuntu not in tested distros - Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop - VAR ${PARALLEL_TEST_ID}= CPT005.201 scope=TEST - Add Parallel Test Skip Condition not ${TESTS_IN_UBUNTU_SUPPORT} tests in Ubuntu not supported + # Load Ubuntu Add Parallel Test Skip Condition + ... ${CPF_LOAD_ID}.201 + ... not ${TESTS_IN_UBUNTU_SUPPORT} + ... tests in Ubuntu not supported + Add Parallel Test Skip Condition ${CPF_LOAD_ID}.201 ... '${ENV_ID_UBUNTU}' not in ${TESTED_LINUX_DISTROS} ... Ubuntu not in tested distros - Add Parallel Test Skip Condition ${LAPTOP_PLATFORM} The Platform is a Laptop Check CPU Frequencies Not Stuck [Documentation] Check if a list of CPU frequencies shows them being stuck diff --git a/lib/parallel-testing.robot b/lib/parallel-testing.robot index 74edf31e64..d3b2572efa 100644 --- a/lib/parallel-testing.robot +++ b/lib/parallel-testing.robot @@ -5,7 +5,6 @@ Library DateTime *** Variables *** &{PARALLEL_TESTS}= &{EMPTY} -${PARALLEL_TEST_ID}= ${EMPTY} *** Keywords *** @@ -13,7 +12,6 @@ Init Parallel Testing [Documentation] Initializes variables used by the parallel testing library. ... Needs to be called before any other library keyword is called. VAR &{PARALLEL_TESTS}= &{EMPTY} scope=GLOBAL - VAR ${PARALLEL_TEST_ID}= ${EMPTY} scope=GLOBAL Wait Until Time [Documentation] Sleeps until given DateTime is reached. @@ -28,11 +26,10 @@ Add Parallel Test Skip Condition [Documentation] Creates a parallel test context if does not exist. ... Verifies a skip condition immediately and saves the skip state and ... a skip reason if the skip condition is true. - [Arguments] ${condition} ${skip_reason} - # [Globals] ${TESTS_TO_RUN} ${TEST_ID} - ${current_details}= Get From Dictionary ${PARALLEL_TESTS} ${PARALLEL_TEST_ID} default=${FALSE} + [Arguments] ${test_id} ${condition} ${skip_reason} + ${current_details}= Get From Dictionary ${PARALLEL_TESTS} ${test_id} default=${FALSE} IF not ${current_details} - VAR &{support_details}= test=${PARALLEL_TEST_ID} run=${TRUE} skip_reason=supported + VAR &{support_details}= test=${test_id} run=${TRUE} skip_reason=supported ELSE VAR ${support_details}= ${current_details} END @@ -40,7 +37,7 @@ Add Parallel Test Skip Condition IF ${run} and (${condition}) Set To Dictionary ${support_details} run=${FALSE} skip_reason=${skip_reason} END - Set To Dictionary ${PARALLEL_TESTS} ${PARALLEL_TEST_ID}=${support_details} + Set To Dictionary ${PARALLEL_TESTS} ${test_id}=${support_details} Check Parallel Test Supported [Documentation] Returns TRUE/FALSE if the given parallel test case will From d3ef4214550fc91340117dd3186a198387cf86ca Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Filip=20Go=C5=82a=C5=9B?= Date: Wed, 22 Oct 2025 08:38:49 +0200 Subject: [PATCH 13/13] parallel-testing.robot: Handle get/set of unitialized tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: Filip Gołaś --- lib/parallel-testing.robot | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/lib/parallel-testing.robot b/lib/parallel-testing.robot index d3b2572efa..89bad80f4d 100644 --- a/lib/parallel-testing.robot +++ b/lib/parallel-testing.robot @@ -135,7 +135,8 @@ Skip If Parallel Test Wont Be Run Regex Get Parallel Test Value [Documentation] Returns a value for a given key in a parallel test context. [Arguments] ${test} ${key} - ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test} + ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test} ${None} + IF $test is ${None} RETURN ${None} ${value}= Get From Dictionary ${test} ${key} ${EMPTY} RETURN ${value} @@ -148,7 +149,11 @@ Get Parallel Test Outputs Set Parallel Test Value [Documentation] Sets the value of a given key for a parallel test context. [Arguments] ${test_id} ${key} ${value} - ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test_id} + ${test}= Get From Dictionary ${PARALLEL_TESTS} ${test_id} ${None} + IF $test is ${None} + Log Test ${test_id} was not initialized + RETURN + END Set To Dictionary ${test} ${key}=${value} Set To Dictionary ${PARALLEL_TESTS} ${test_id}=${test}