[CI] Integration tests regrouping #6349

pazone · 2024-12-17T12:36:56Z

What does this PR do?

Extracts 3 new test groups that are executed separately:

standalone-upgrade
fleet-endpoint-security
install-uninstall (our favorite)

Now the longest running group is fleet-endpoint-security (42 minutes)

The TestRepeatedInstallUninstall on PRs will have 50 iterations instead of 100. For the main and release branches the iterations number will stay 100.

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in ./changelog/fragments using the changelog tool
I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Relates https://github.com/elastic/ingest-dev/issues/4216

Questions to ask yourself

How are we going to support this in production?
How are we going to measure its adoption?
How are we going to debug this?
What are the metrics I should take care of?
...

testing/integration/install_test.go

swiatekm · 2024-12-18T12:03:20Z

testing/integration/install_test.go

+	if os.Getenv("BUILDKITE_PULL_REQUEST") != "" {
+		return 50
+	}


This deserves a comment in my opinion.

rowlandgeoff · 2024-12-18T23:50:02Z

.buildkite/bk.integration.pipeline.yml

@@ -40,8 +40,11 @@ steps:
        matrix:
          - default
          - fleet
+          - fleet-endpoint-security


The ordering in these matrix lists looks arbitrary. Is there a reason for the ordering, or should we have them sorted alphabetically?

Honestly, there are no ordering rules. Just placed fleet groups together

Are you going to sort them? If the order doesn't matter, I think it'd be a better user experience when engineers are scrolling through the results of a BK job to have them alphabetical. Plus it would make these lists more easily maintainable (until we figure out making the groups/matrix dynamic).

blakerouse · 2025-02-12T15:08:24Z

This seems to be better and no longer getting the weird enrollment race conditions that used to be present. The change I made was just an improvement to an error message in hopes to track this down. That error is no longer occurring on many tests like it was before. Does seem to have one test related to windows still failing, giving this another clean run to see if I can track that one down.

elastic-sonarqube · 2025-02-12T16:00:03Z

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube

blakerouse · 2025-02-12T21:36:31Z

This is looking like it is now passing, I think this is ready to be merged. I recently did some changes to enrollment in Fleet Server that might relate to this being less likely to cause issues with enrollment. Whatever the case was it is no longer an issue with this PR.

Needs an approval from @elastic/ingest-eng-prod to allow merge.

jlind23 · 2025-02-13T07:03:49Z

@pkoutsovasilis heads up, this is going to be merged.

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) # Conflicts: # pkg/testing/tools/fleettools/fleet.go # testing/integration/install_test.go

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f)

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) # Conflicts: # pkg/testing/tools/fleettools/fleet.go # testing/integration/install_test.go

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f)

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co>

* [CI] Integration tests regrouping (#6349) * TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) # Conflicts: # pkg/testing/tools/fleettools/fleet.go # testing/integration/install_test.go * Resolved Conflicts --------- Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co>

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co>

* [CI] Integration tests regrouping (#6349) * TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) # Conflicts: # pkg/testing/tools/fleettools/fleet.go # testing/integration/install_test.go * Resolved install_test.go * Resolved conflicts --------- Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co> Co-authored-by: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>

* TestRepeatedInstallUninstall optimisation * cleanup * cleanup * cleanup * cleanup * TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group * Extracted standalone-upgrade group * Commented iterations for TestRepeatedInstallUninstall * Reordered groups in pipeline matrix * Fix error message to only include agents on the same policy. --------- Co-authored-by: Denis <denis.rechkunov@elastic.co> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> (cherry picked from commit 37be67f) Co-authored-by: Pavel Zorin <pavel.zorin@elastic.co>

pazone added 2 commits December 17, 2024 13:17

TestRepeatedInstallUninstall optimisation

f017e54

cleanup

8490bf7

pazone added enhancement New feature or request backport-skip skip-changelog Team:Ingest-EngProd labels Dec 17, 2024

mergify bot assigned pazone Dec 17, 2024

rdner reviewed Dec 17, 2024

View reviewed changes

testing/integration/install_test.go Outdated Show resolved Hide resolved

testing/integration/install_test.go Outdated Show resolved Hide resolved

pazone added 2 commits December 17, 2024 13:43

cleanup

f1f9aec

cleanup

ebc0c49

rdner approved these changes Dec 17, 2024

View reviewed changes

pazone added 2 commits December 17, 2024 14:15

cleanup

2105279

TestInstallDefendWithMTLSandEncCertKey to fleet-install-defend group

a8df411

pazone marked this pull request as ready for review December 17, 2024 22:02

pazone requested review from a team as code owners December 17, 2024 22:02

pazone requested review from swiatekm and pchila December 17, 2024 22:02

Extracted standalone-upgrade group

89fd608

pazone marked this pull request as draft December 17, 2024 22:31

pazone changed the title ~~[CI] TestRepeatedInstallUninstallFleet optimisation~~ [CI] Integration tests regrouping Dec 17, 2024

pazone added backport-8.x Automated backport to the 8.x branch with mergify backport-8.16 Automated backport with mergify backport-8.17 Automated backport with mergify and removed backport-skip labels Dec 18, 2024

pazone marked this pull request as ready for review December 18, 2024 10:31

swiatekm approved these changes Dec 18, 2024

View reviewed changes

rdner added 2 commits December 18, 2024 17:44

Merge branch 'main' into ci_integration_test_regroup

5051e4a

Merge branch 'main' into ci_integration_test_regroup

a1b5b95

rowlandgeoff reviewed Dec 18, 2024

View reviewed changes

Merge branch 'main' into ci_integration_test_regroup

85cc127

blakerouse approved these changes Feb 12, 2025

View reviewed changes

jlind23 approved these changes Feb 13, 2025

View reviewed changes

pazone merged commit 37be67f into main Feb 13, 2025
18 checks passed

pazone deleted the ci_integration_test_regroup branch February 13, 2025 11:31

pazone added backport-9.0 Automated backport to the 9.0 branch and removed backport-9.0 Automated backport to the 9.0 branch labels Feb 13, 2025

This was referenced Feb 13, 2025

[8.16](backport #6349) [CI] Integration tests regrouping #6854

Merged

[8.x](backport #6349) [CI] Integration tests regrouping #6855

Merged

mergify bot mentioned this pull request Feb 13, 2025

[8.17](backport #6349) [CI] Integration tests regrouping #6856

Merged

7 tasks

This was referenced Feb 13, 2025

[8.18](backport #6349) [CI] Integration tests regrouping #6857

Merged

[9.0](backport #6349) [CI] Integration tests regrouping #6858

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[CI] Integration tests regrouping #6349

[CI] Integration tests regrouping #6349

Uh oh!

pazone commented Dec 17, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

swiatekm Dec 18, 2024

Uh oh!

rowlandgeoff Dec 18, 2024

Uh oh!

pazone Dec 19, 2024

Uh oh!

rowlandgeoff Dec 19, 2024

Uh oh!

blakerouse commented Feb 12, 2025

Uh oh!

elastic-sonarqube bot commented Feb 12, 2025

Uh oh!

blakerouse commented Feb 12, 2025

Uh oh!

jlind23 commented Feb 13, 2025

Uh oh!

Uh oh!

Uh oh!

[CI] Integration tests regrouping #6349

[CI] Integration tests regrouping #6349

Uh oh!

Conversation

pazone commented Dec 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Why is it important?

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

Uh oh!

Uh oh!

Uh oh!

swiatekm Dec 18, 2024

Choose a reason for hiding this comment

Uh oh!

rowlandgeoff Dec 18, 2024

Choose a reason for hiding this comment

Uh oh!

pazone Dec 19, 2024

Choose a reason for hiding this comment

Uh oh!

rowlandgeoff Dec 19, 2024

Choose a reason for hiding this comment

Uh oh!

blakerouse commented Feb 12, 2025

Uh oh!

elastic-sonarqube bot commented Feb 12, 2025

Quality Gate passed

Uh oh!

blakerouse commented Feb 12, 2025

Uh oh!

jlind23 commented Feb 13, 2025

Uh oh!

Uh oh!

Uh oh!

pazone commented Dec 17, 2024 •

edited

Loading