Allow pinning the snapshot version used in integration tests to a specific build ID

[cmacknz]

• Relates Automate delaying the stack version bump in integration tests in the release process #3440

We recently had a problem where all of our endpoint-security integration tests were failing because of a bug in endpoint-security itself. One solution to this problem would have been to skip all of the endpoint-security tests until the problem was resolved, but this would have allowed us to break support for endpoint-security without noticing sometime later.

A better solution is to allow pinning the versions (of the stack deployment and each individual binary we depend on) we use in integration tests to a specific snapshot version. This would have allowed us to pin our dependencies to a snapshot version before the problem was introduced, allowing us to keep executing tests without issue.

We can initially continue to have main test against the latest available version and only use version pinning when we need to avoid a known issue.

We can later evaluate a model where main is always pinned to a known working snapshot version, with the version being automatically advanced via a PR to ensure the tests continue to pass as dependencies are updated. This will do a much better job of allowing us to isolate failures introduce in test dependencies, but will also allow us to ignore them for longer.

For reference each available snapshot has both a latest (for example 8.13.0-SNAPSHOT) and specific build IDs (for example 0bexikfg). To see the latest snapshots for a specific version see https://artifacts-api.elastic.co/v1/search/8.13.0-SNAPSHOT/elastic-agent which at the time of writing indicates the latest snapshot build ID is 0bexikfg available from https://snapshots.elastic.co/8.13.0-0bexikfg/downloads/beats/elastic-agent/elastic-agent-8.13.0-SNAPSHOT-darwin-aarch64.tar.gz

Acceptance Criteria

• The snapshot build ID to use when creating stack deployments can be pinned when using the AGENT_STACK_VERSION variable to mage integration:test and related targets.
• The snapshot version to use when downloading dependent binaries with mage package can be controlled using a new environment variable.
• Both the stack and agent binary dependency versions can easily be overridden, similar to our existing agent package version override:

  elastic-agent/.buildkite/scripts/steps/integration_tests.sh

  Lines 11 to 20 in b664149

  	# Override the agent package version using a string with format <major>.<minor>.<patch>
  	# NOTE: use only after version bump when the new version is not yet available, for example:
  	# OVERRIDE_AGENT_PACKAGE_VERSION="8.10.3" otherwise OVERRIDE_AGENT_PACKAGE_VERSION="".
  	OVERRIDE_AGENT_PACKAGE_VERSION="8.12.0"
  	if [[ -n "$OVERRIDE_AGENT_PACKAGE_VERSION" ]]; then
  	OVERRIDE_TEST_AGENT_VERSION=${OVERRIDE_AGENT_PACKAGE_VERSION}"-SNAPSHOT"
  	else
  	OVERRIDE_TEST_AGENT_VERSION=""
  	fi

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)