Ess terraform + run integration tests on BK agents (backport #5113)

[mergify]

What does this PR do?

A dedicated pipeline that runs Ubuntu integration tests directly on Buildkite runners without OGC. Currently it's not triggered by any of Github events.
Sudo tests build example

• Improves groupping transparency. We can now immediately see long-running groups;
• Adds the ability to rerun failed groups instead of the entire pipeline build;
• Introduces a simple terraform configuration for ESS stack instead of a complex stack provisioner;
• Uses prebuilt VM images platform-ingest-elastic-agent-ubuntu-2204. All required packages are pre-installed and we don't need to install them in runtime;
• Adds the ability to use all the gobld-supported CSPs and VMs incl. ARM64 and MacOS.

Shared ESS stack and test reruns

The Start ESS stack for integration tests spins up an ESS stack that is used by following integration tests. When the tests are complete, the stack will be destroyed regardless of the test results. When we restart a single test group, it will start a dedicated stack and destroy it when the tests are complete.

This type of pipeline is supposed to replace the existing single-step approach. Further upcoming changes a listed in the the meta-issue.

Why is it important?

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in ./changelog/fragments using the changelog tool
• I have added an integration test or an E2E test

Disruptive User Impact

How to test this PR locally

Related issues

Questions to ask yourself

• How are we going to support this in production?
• How are we going to measure its adoption?
• How are we going to debug this?
• What are the metrics I should take care of?
• ...

---

This is an automatic backport of pull request #5113 done by [Mergify](https://mergify.com).

[mergify]

Cherry-pick of 7aa8bb2 has failed:

On branch mergify/bp/8.16/pr-5113
Your branch is up to date with 'origin/8.16'.

You are currently cherry-picking commit 7aa8bb222f.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both added:      .buildkite/bk.integration.pipeline.yml
	both added:      .buildkite/scripts/steps/ess.sh
	both added:      .buildkite/scripts/sudo-integration-tests.sh

no changes added to commit (use "git add" and/or "git commit -a")

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

[mergify]

This pull request has not been merged yet. Could you please review and merge it @oakrizan? 🙏

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

[mergify]

This pull request has not been merged yet. Could you please review and merge it @oakrizan? 🙏

[mergify]

This pull request has not been merged yet. Could you please review and merge it @oakrizan? 🙏