Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reconfigure notification sender worker alarms #1181

Merged
merged 4 commits into from
Nov 13, 2023
Merged

Conversation

tkgnm
Copy link
Contributor

@tkgnm tkgnm commented Nov 9, 2023

Why

We have many alarms related to this stack in a permanent state of alarm. This is because they're configured to alarm if no invocations in 6 minutes. However....

  • The editions app only sends one notification a day at 4am
  • As for the live app, it's not uncommon for a notification not to be sent by editors for over 6 minutes.
  • We also have alarms going off in CODE - the CODE stack seems to send a test notification every hour, not every 6 minutes

What does this change?

This change reconfigures our notification sender worker alarms.

  • Editions apps will only alarm if no invocations for one day
  • Other alarms will only alarm if no invocations for one hour
  • Enables alarms being sent to our P&E/Apps/Server Alerts channel

These changes should mean our alarms are more useful and enable us to act on them in case of something going wrong by subscribing them to our alerts channel.

Testing

To test the alerts to our channel, I changed the period to 1 minute and deployed to CODE to ensure we would get alerts.

image

@tkgnm tkgnm force-pushed the alarm-on-prod-only branch 2 times, most recently from 912133b to 895f9c3 Compare November 10, 2023 11:10
@tkgnm tkgnm changed the title Alarm on prod only Reconfigure notification sender worker alarms Nov 10, 2023
Comment on lines -3 to 4
exports[`The MobileAppsRendering stack matches the snapshot 1`] = `
exports[`The Sender Worker stack matches the snapshot 1`] = `
Object {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong test name! Probably a copy/paste job.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch

Update snap

Disable alarm on CODE

increase alarm period

Once a day for editions as they only send one notification a day. Once every 30 minutes for others as we often times go more than 30 minutes without sending a notification.

Reinstate on CODE

remove line

test with short period
@tkgnm tkgnm marked this pull request as ready for review November 10, 2023 11:41
@tkgnm tkgnm requested a review from a team November 10, 2023 11:44
Copy link
Contributor

@waisingyiu waisingyiu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link

@groakland groakland left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tkgnm tkgnm merged commit 795654d into main Nov 13, 2023
11 checks passed
@tkgnm tkgnm deleted the alarm-on-prod-only branch November 13, 2023 07:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants