Add flag to allow the agent to opt-in to attempting to upgrade from staging.elastic.co

[cmacknz]

We see integration test failures when the first build candidate for a new release is first made available because the agent does not know to look for versions at staging.elastic.co (for example the final 8.11.0 BC is hosted at https://staging.elastic.co/8.11.0-d0c3d0fd/downloads/beats/elastic-agent/elastic-agent-8.11.0-darwin-aarch64.tar.gz).

The problem is that once a build candidate becomes available it is listed in https://artifacts-api.elastic.co/v1/versions, which is what our integration tests used to determine the set of upgradable versions for testing. We currently have non-trivial logic to try to infer if the version we are considering is a build candidate and skip it. This mostly works but has trouble handling the case where a new build candidate for patch version of the current minor is available. For example main is 8.13.0, the 8.12.0 branch exists with no BC, but an 8.11.3 BC is available.

It will be much simpler to add a way to configure the agent to attempt the download from staging.elastic.co in addition to artifacts.elastic.co and snapshot.elastic.co. Note that we do not want this enabled by default, because we do not want a temporary failure to download from an official release at artifacts.elastic.o to fall back to a build candidate at staging.elastic.co. This risk does not exist for snapshots because the version name convention is different.

The new configuration flag should exist as an agent.download setting and be set by default in our integration tests so that the time period where build candidates exist is handled seamlessly.

elastic-agent/elastic-agent.reference.yml

Lines 88 to 91 in 7e80290

	# agent.download:
	# # source of the artifacts, requires elastic like structure and naming of the binaries
	# # e.g /windows-x86.zip
	# sourceURI: "https://artifacts.elastic.co/downloads/beats/"

We can include something like an allow_staging flag that adds staging.elastic.co to the composed downloader: https://github.com/cmacknz/elastic-agent/blob/e93010f40c8cd20b1a63cd678a353fb7abba017b/internal/pkg/agent/application/upgrade/step_download.go#L180

It may be preferable to introduce a source_uris parameter that allows specifying multiple possible download sources without needing to embed the staging URL in the agent source like we do for snapshots.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

Solved by moving the agent versions to a file.

Will also be more explicitly solved by https://github.com/elastic/ingest-dev/issues/3344