Ess terraform + run integration tests on BK agents (#5113)

* ESS by terraforms

* Use bk runners

* added state to artifacts

* Add common.sh to scripts

* Add depends_on

* Inlined paclage variables

* tmp disbale unit tests

* tmp disbale unit tests

* install tf

* install tf

* install tf

* install tf

* fix ec creds

* Added auto-approve

* Test run

* package again

* package again

* package again

* package again

* package

* package

* added mage build:testBinaries

* Install mage

* Install mage

* Install mage

* Install mage

* Install mage

* fix go

* Merged

* Merged

* fix

* fix

* Fix tests

* Debug

* Use custom image

* New suod tests

* Test run

* Test run

* Test run

* Debug

* Debug

* Debug

* Debug

* Debug

* Cleanup

* Using subshell in newgrp

* Using subshell in newgrp

* Test new custom VM image

* Test new custom VM image

* Set terraform version

* Cleanup

* terraform 1.9.3

* terraform 1.9.3

* Debug

* Explicit image version

* Explicit image version

* Explicit image version

* Sudo tests

* +x

* Try mage integration:local

* Use gotestsum

* Use gotestsum

* Run sudo tests

* Run sudo tests

* Debug

* Debug

* Debug

* Debug

* Debug

* Debug

* Debug

* Debug

* Increased test timeout

* Groupped sudo tests

* Quoted regexps

* BK steps by groups

* BK steps by groups

* BK steps by groups

* BK steps by groups

* fix param propagation

* Debug

* Big run

* reuse EC_API_KEY

* Debug test

* Debug a test

* Revert debug

* Debug

* Html report

* groupping, auto-retry, packaging

* removed reruns

* Reuse the initial ESS stack. Start a new stack for retries

* Fix ess start path

* Fix ess start path

* Fix ess start path

* Fix ess start path

* Debug

* Debug

* Debug

* Fix EC_KEY recovery

* Added lock to artifacts

* Fix artifacts download path

* Fix artifacts download path

* Fix artifacts download path

* Removed unnecessaru build dependencies

* Added build id

* Separated BK integration tests to a dedicated pipeline

* CLeanup

* Applied proposed changes

* Applied proposed changes

* Applied proposed changes

(cherry picked from commit 7aa8bb2)

.buildkite/scripts/steps/ess.sh

@@ -2,7 +2,7 @@
 set -euo pipefail
 
 function ess_up() {
-  echo "~~~ Staring ESS Stack"
+  echo "~~~ Starting ESS Stack"
   local WORKSPACE=$(git rev-parse --show-toplevel)
   local TF_DIR="${WORKSPACE}/test_infra/ess/"
   local STACK_VERSION=$1