tools: Adds a Sonobuoy conformance testing script

[etungsten]

Issue #, if available: N/A

Description of changes:
Adds sonobuoy conformance testing scripts that spins up an EKS cluster
then runs sonobuoy conformance testing, after its done, it retrieves the
results and spins down the EKS cluster.

setup-test-cluster.sh sets up the cluster and generates env file and user data
run-conformance-test.sh consumes the generated env file and user data to launch thar worker nodes and runs sonobuoy conformance tests.
clean-test-cluster.sh cleans up the test cluster specified by the env file.

Borrows several helper functions from the amiize.sh script.

Testing:
Running in us-west-2:

Setup cluster:

$ ./setup-test-cluster.sh --region us-west-2 --cluster-name sono-test
Setting up fresh EKS cluster with eksctl
[ℹ]  eksctl version 0.9.0
[ℹ]  using region us-west-2
[ℹ]  setting availability zones to [us-west-2b us-west-2d us-west-2a]
[ℹ]  subnets for us-west-2b - public:192.168.0.0/19 private:192.168.96.0/19
[ℹ]  subnets for us-west-2d - public:192.168.32.0/19 private:192.168.128.0/19
[ℹ]  subnets for us-west-2a - public:192.168.64.0/19 private:192.168.160.0/19
[ℹ]  nodegroup "ng-fe807768" will use "ami-05d586e6f773f6abf" [AmazonLinux2/1.14]
[ℹ]  using Kubernetes version 1.14
[ℹ]  creating EKS cluster "sono-test" in "us-west-2" region
[ℹ]  will create 2 separate CloudFormation stacks for cluster itself and the initial nodegroup
[ℹ]  if you encounter any issues, check CloudFormation console or try 'eksctl utils describe-stacks --region=us-west-2 --cluster=sono-test'
[ℹ]  CloudWatch logging will not be enabled for cluster "sono-test" in "us-west-2"
[ℹ]  you can enable it with 'eksctl utils update-cluster-logging --region=us-west-2 --cluster=sono-test'
[ℹ]  Kubernetes API endpoint access will use default of {publicAccess=true, privateAccess=false} for cluster "sono-test" in "us-west-2"
[ℹ]  2 sequential tasks: { create cluster control plane "sono-test", create nodegroup "ng-fe807768" }
[ℹ]  building cluster stack "eksctl-sono-test-cluster"
[ℹ]  deploying stack "eksctl-sono-test-cluster"
[ℹ]  building nodegroup stack "eksctl-sono-test-nodegroup-ng-fe807768"
[ℹ]  --nodes-min=0 was set automatically for nodegroup ng-fe807768
[ℹ]  --nodes-max=0 was set automatically for nodegroup ng-fe807768
[ℹ]  deploying stack "eksctl-sono-test-nodegroup-ng-fe807768"
[✔]  all EKS cluster resources for "sono-test" have been created
[✔]  saved kubeconfig as "/home/ANT.AMAZON.COM/etung/.kube/config"
[ℹ]  adding identity "arn:aws:iam::722737851570:role/eksctl-sono-test-nodegroup-ng-fe8-NodeInstanceRole-1R4WLQLMF3FMT" to auth ConfigMap
[ℹ]  kubectl command should work with "/home/ANT.AMAZON.COM/etung/.kube/config", try 'kubectl get nodes'
[✔]  EKS cluster "sono-test" in "us-west-2" region is ready
Writing kubeconfig for sono-test to sono-test-config
[ℹ]  eksctl version 0.9.0
[ℹ]  using region us-west-2
[✔]  saved kubeconfig as "sono-test-config"
Updating the CNI plugin version
daemonset.extensions/aws-node patched
Generating userdata file for launching Thar worker nodes
Getting instance profile for use in instance launches
Setting up security groups
Generating env file for launching Thar worker nodes
Finished setting up test EKS cluster.
Userdata file: sono-test-user-data.toml
Env file: sono-test.env

Run conformance tests:

$ ./run-conformance-test.sh --cluster-env-file sono-test.env --node-ami ami-07245e9300b9290c1 --instance-type m5.large
Launching 3 Thar worker nodes
Waiting for all Thar worker nodes to become 'Ready' in sono-test cluster
ready: 1
ready: 1
ready: 2
ready: 3
Starting Sonobuoy Kubernetes conformance test! Test may take up to 60 minutes to finish
INFO[0000] created object                                name=sonobuoy namespace= resource=namespaces
INFO[0000] created object                                name=sonobuoy-serviceaccount namespace=sonobuoy resource=serviceaccounts
INFO[0000] created object                                name=sonobuoy-serviceaccount-sonobuoy namespace= resource=clusterrolebindings
INFO[0000] created object                                name=sonobuoy-serviceaccount namespace= resource=clusterroles
INFO[0000] created object                                name=sonobuoy-config-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object                                name=sonobuoy-plugins-cm namespace=sonobuoy resource=configmaps
INFO[0000] created object                                name=sonobuoy namespace=sonobuoy resource=pods
INFO[0000] created object                                name=sonobuoy-master namespace=sonobuoy resource=services
         PLUGIN     STATUS   RESULT   COUNT
            e2e   complete   passed       1
   systemd-logs   complete   passed       1
   systemd-logs   complete                2

Sonobuoy has completed. Use `sonobuoy retrieve` to get results.
Plugin: e2e
Status: passed
Total: 3586
Passed: 204
Failed: 0
Skipped: 3382

Plugin: systemd-logs
Status: passed
Total: 3
Passed: 3
Failed: 0
Skipped: 0
Sonobuoy test results available at sono-test-conformance-test-results/201911292327_sonobuoy_89e4a4ff-1e63-4ec4-9500-307a9c73cdbb.tar.gz
Cleaning up Sonobuoy namespace, may take up to 20 minutes
INFO[0000] deleted                                       kind=namespace namespace=sonobuoy
INFO[0000] deleted                                       kind=clusterrolebindings
INFO[0000] deleted                                       kind=clusterroles
Cleaning up Thar worker node instances
TERMINATINGINSTANCES	i-0270f18beddc66746
CURRENTSTATE	32	shutting-down
PREVIOUSSTATE	16	running
TERMINATINGINSTANCES	i-0d4343f1475f3c379
CURRENTSTATE	32	shutting-down
PREVIOUSSTATE	16	running
TERMINATINGINSTANCES	i-0ba340884d9bd4f3b
CURRENTSTATE	32	shutting-down
PREVIOUSSTATE	16	running

Clean up cluster:

$ ./clean-up-test-cluster.sh --cluster-env-file sono-test.env
Removing security group dependencies.
Deleting the test cluster.
[ℹ]  eksctl version 0.9.0
[ℹ]  using region us-west-2
[ℹ]  deleting EKS cluster "sono-test"
[✔]  kubeconfig has been updated
[ℹ]  cleaning up LoadBalancer services
[ℹ]  2 sequential tasks: { 2 parallel sub-tasks: { cleanup for nodegroup "ng-f98bc379", delete nodegroup "ng-f98bc379" }, delete cluster control plane "sono-test" [async] }
[ℹ]  trying to cleanup dangling network interfaces
[ℹ]  will delete stack "eksctl-sono-test-nodegroup-ng-f98bc379"
[ℹ]  waiting for stack "eksctl-sono-test-nodegroup-ng-f98bc379" to get deleted
[ℹ]  will delete stack "eksctl-sono-test-cluster"
[✔]  all cluster resources were deleted
Deleting env file, userdata file, and kubeconfig file
Clean up done.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[tjkirch]

Talked with @etungsten a bit outside of the PR. I think we should aim for separating the setup and run portions. That way developers can leave the infrastructure and just do a quick test run when desired, and CI can spin up fresh as needed.

[etungsten]

Split the monolithic script into three parts: setup-test-cluster.sh, run-conformance-test.sh and clean-up-test-cluster.sh.
Adds a README to help clarify usage.

[tjkirch]

Would you please update the testing for the new script separation? (I don't think we need to see two regions; your assurance that it works the same is enough there :))

[etungsten]

Updated testing description.
Minor fixes to clean-up-test-cluster.sh

[jahkeup]

Does --nodes 0 here cause eksctl to setup the nodegroup resources without starting nodes? Does it create an ASG that can later be scaled?

[etungsten]

Yes it does! Yes it does!

[jahkeup]

Chatted with @etungsten offline: the ASG aspect won't be useful to us until we have an eksctl in place that has the desired CNI plugin configured (for aws-node) and the TOML userdata generation.

In talking with @tjkirch about how #363 could help reduce scope, I think we came to an understanding that it would be reasonable to rely on a patched version if the upstream gave positive indications that they'd accept thar node-ami-family contributions. Please correct me if I misunderstood @tjkirch 👍

[tjkirch]

In talking with @tjkirch about how #363 could help reduce scope, I think we came to an understanding that it would be reasonable to rely on a patched version if the upstream gave positive indications that they'd accept thar node-ami-family contributions. Please correct me if I misunderstood @tjkirch

I think that's a minimum requirement, but we should still discuss with the team to make sure we'd want to suggest that (1) users need a patched tool, or (2) we have branching instructions.

[etungsten]

Removes setting up a completely new instance profile.
Patching the ConfigMap is no longer necessary as a result.
Launch Thar nodes using the nodegroup instance profile set up by eksctl.

[tjkirch]

Partial review - I think GitHub is preventing me from responding to other comments because I have these pending..?

[jahkeup]

Looking good! There's a few scripty things I had questions on as well as sonobouy result outputs that I want to understand a bit more.

[etungsten]

Addresses @tjkirch 's and @jahkeup 's comments.
Testing changes

Force push below fixes missing newline at the end of file.

[etungsten]

Redirect hash output to /dev/null in favour of more explicit error messages.

[tjkirch]

GitHub merged force-pushes; here's the full diff from the changes etungsten made 2019-11-27:

https://github.com/amazonlinux/PRIVATE-thar/compare/231ef76..d418c97

[etungsten]

Addresses @tjkirch 's comments:

• Removed mapfile
• Moved deleting the env files to after cluster deletion completion
• Capture kubectl --kubeconfig ... and sonobuoy --kubeconfig .. in variables
• Specify source group ids for the ingress and egress security group rules
• Added set -o pipefail to where things are being piped
• Got rid of ec2 wait
• Moved sonobuoy delete to after instance clean up

Testing again to make sure everything still works.

[etungsten]

Addresses @tjkirch 's comments.

[etungsten]

Removes the added security group egress and security group ingress rules during clean up.
It created interdependency between cluster security groups which causes CloudFormation deletion failures.

[etungsten]

Addresses @tjkirch's comment

[jahkeup]

Looks good! Let's get going with this and revisit if needed. 👍