Basic Capture Metrics, Scaling, and Alarming

[chelma]

Description

This task is to provide CloudWatch Metrics/Alarming on the utilization of the Capture Nodes and OpenSearch domain and to set up scaling of the Capture Nodes on those metrics, if this is not already configured.

OpenSearch domains are not easy to set up automatic scaling on (the docs indicate a manual operation is expected [1], and there does not appear to be an autoscaling feature). While it is theoretically possible to automate scaling using metrics and the UpdateDomainConfig API [2], it would require substantial effort. Therefore, we will set up metrics/alarming for the domain but leave scaling as a manual operation for the user.

[1] https://aws.amazon.com/premiumsupport/knowledge-center/opensearch-scale-up/
[2] https://docs.aws.amazon.com/opensearch-service/latest/APIReference/API_UpdateDomainConfig.html

Acceptance Criteria

• Capture Nodes should scale on utilization (massive loads not considered)
• OpenSearch Domain scaling will be manual
• Metrics and reasonable CloudWatch Alarms should be present in console.