hcdiag kubernetes compatible container image for support request bundles and troubleshooting

[natemollica-nm]

What problem would you like to address?
The current flexibility of running hcdiag locally on a mac holding the hcdiag binary may not be ideal for most customers and troubleshooting steps aren't so concise for consul-k8s. i.e.,

• Download hcdiag binary to local machine (determine how to obtain the latest)
• Download consul-k8s cli tool to local machine
• Port forward consul on kubernetes to localhost:8500 (or 8501)
• Build custom hcl configuration with consul-k8s cli commands to retrieve consul-k8s specific configuration items

With the above steps, capturing information from multiple clusters (i.e., federated/peered, etc.) can become somewhat convoluted with port-forwarding in this case with kubernetes during outage type scenarios.

Who is the audience for this request?

• Consul Support Engineering
• Customers maintaining service discovery and service mesh use-cases with Consul

Which HashiCorp products does this affect?

• consul

What would the ideal solution look like?

• Addition of helm configuration overrides for creating and maintaining a kubernetes job that runs hcdiag (and optionally with a config file)
• Ability to configure this job to run in scenarios such as outages or loss of cluster leadership (i.e., failed consul-server readiness probes).

What alternatives have you considered?

• Updating the docs README files to include manual steps to run hcdiag on consul-k8s for any bug/outage scenarios.

How does this impact you?

• Simplifies and shortens the timeline for diagnosing and understanding the scope of:
  • The customer's environment configuration
  • The issue the customer is experiencing

What are the customer data privacy implications for this request?

• Consul ACL Tokens with snapshot/catalog/operator write permissions would be required to be set either environmentally within the kubernetes pod or on the hcdiag container.
• Potentially creating a kubernetes secret that contains the hcdiag information bundle that could contain cluster ip information

Are there any deadlines?

• No deadlines currently, but customers have expressed frustration that this isn't already implemented in the release of consul-k8s

Is there any additional context?

Currently support is making effort to retrieve customer environment configuration specifics for those customers that want enterprise level of support, and something like this feature would improve our capability of quickly onboarding those customers.