Use bigger EC2 instance for notification #1212
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lindseydew
approved these changes
Mar 26, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this change?
Editorials reported that they sent a breaking news notification via ed tools but no notifications were received. The notification record was not found on Ophan dashboard either.
After investigation, it was found that the breaking news tool sent a HTTP request to our notification endpoint but received a 503 exception.
The cloudwatch metric (ELB 5xx) for the load balancer of the notification API suggested that a 5xx response had been served from the load balancer at that time. One of the EC2 instances was also terminated due to health check failure around that minute.
We noticed that EC2 instances of notification service failed health check from time to time. The application logs of an unhealthy instance did not show any exception or error message, but the syslog of the OS indicated that there was out-of-memory error on OS process level.
I believe that the JVM of the service ran out of system memory when it was expanding its heap, given that we are using
t4g.microinstance which has 1G memory only but the notification service and the AWS kinesis agent have a max heap size of 256M and 512M respectively.This PR changes the cloudformation stack to use bigger EC2 instance,
t4g.small, which has 2G memory.How to test
I applied this bigger EC2 instance on CODE and the instance has stayed healthy for more than 1 day. It may be good to apply it on PROD too, and see if the problem with instances becoming unhealthy ceases to exist.
How can we measure success?
No instances become unhealthy.