Investigate unauthorised-api-calls alarms and take action to reduce noise

[richgreen-moj]

User Story

As a MP Engineer
I want to investigate the cause behind the large amount of unauthorised-api-calls alarm triggers
So that I can reduce the noise in the #modernisation-platform-low-priority-alarms slack channel

Value / Purpose

The aim of this ticket is to review the unauthorised-api-calls alarm which is being triggered in many different accounts and try to understand why it is triggered so often. Then next steps are to decide on the best course of action which could include resolving the root cause or simply altering the threshold of the alarm.

Ideally we want to be able to proactively monitor the #modernisation-platform-low-priority-alarms slack channel but there is currently too much noise for it to be reasonable to do so.

Context / Background

https://mojdt.slack.com/archives/C013RM6MFFW/p1738592534773869 << Slack thread for extra context

https://moj-digital-tools.pagerduty.com/service-directory/PYDTWVM << PagerDuty incidents raised by the alarm

You can review the cloudwatch logs to see what is triggering the alarms in each account and see if there are any common issues. This could just be as simple as members not giving their workloads the appropriate IAM permissions to access certain resources/services etc. It would be interesting to see if it there are any platform-provisioned assets that are creating the problem as they would be more in our wheel-house to fix.

Useful Contacts

@richgreen-moj

Additional Information

https://docs.aws.amazon.com/securityhub/latest/userguide/cloudwatch-controls.html#cloudwatch-2 << this shows the reason for the alarm being set in the first place (CIS recommendation) ....

CIS recommends that you create a metric filter and alarm unauthorized API calls. Monitoring unauthorized API calls helps reveal application errors and might reduce time to detect malicious activity.

Definition of Done

• Review unauthorised-api-calls incidents and associated cloudwatch logs to search for any common patterns etc.
• Based on findings raise issues to fix underlying issues (or just fix if there's a simple win)
• Consider altering the threshold for the alarm to a more appropriate value