Alert somehow when there are cron failures in managed cloud

[roperzh]

Problem

Cron failures in managed cloud do not report to the #help-p1 or #help-p2 channels, which means we're not aware of them unless issues are reported by the customer.

Solution

High-level: add a new errors column to the cron_stats table, populate this with any errors that occur during a cron run if they bubble up to the job runner level (i.e. they cause the job to fail) and then use the existing monitor Lambda to report on these errors hourly.

Details

On the server side:

• Add errors field to cron_stats table (json DEFAULT NULL)
• Add errors var to Schedule struct to collect errors from jobs
  • Errors should be mapped by job, since a schedule can run several jobs
• In RunAllJobs, collect err from job (same place we currently log) into new errors var
• Update Schedule.updateStats and CronStats.UpdateCronStats to accept errors argument
• If provided, update errors field of cron_stats table

On the monitor side:

• Add new SQL query to look for all completed schedules since last run with non-null errors
  send SNS with job ID, name, errors
• “Since last run” could be just now - monitor interval (i.e. in the last hour), but ideally we query AWS for last successful run of the Lambda so we don't miss anything if the Lambda stalls or fails for some reason.
• Add new SNS topic to Terraform to allow sending messages to help-p2 since these are not likely to require immediate action.

Other notes:

• Errors will continued to logged to Cloudwatch as they happen (with “cron” key in the log)
• We had discussed a new failed status for jobs, but now that I know that a schedule can have multiple jobs so that some may error and some not, I don’t think this would be as helpful.

[roperzh]

cc: @lukeheath @georgekarrv

[lukeheath]

@roperzh Thanks for submitting this and good idea and seems necessary. I'm prioritizing to the drafting board for estimation.

[lukeheath]

@georgekarrv I'm prioritizing this for estimation. If it requires assistance from infra please loop in the necessary folks.

@sharon-fdm Do y'all have any cron job alerts that should be generating #help-p1 alerts? If so, please add them to this issue description so they can be included.

@rfairburn Heads up that you may get some questions about this. My primary goal is just to make sure that any cron alerts that should generate #help-p1 alerts (like SCEP certificate) do generate alerts.

Thanks all!

[rfairburn]

This should be possible, but an entirely different mechanism will need to be added to the terraform monitoring module.

I am thinking the pattern would look like this:

Cloudwatch subscription filter -> lambda function (to process) -> SNS Topic (alert #help-p1 slack)

I'd need to flesh the specifics out, but I think it is definitely possible as long as we have easy-to match patterns that don't have ambiguity in what I am matching.

This would be separate from our existing cron monitoring which just checks specific cron jobs for their completed status in the db.

[sharon-fdm]

@lukeheath I am not aware of any alerts mechanism for any of our cron jobs. @getvictor @mostlikelee @lucasmrod do you know otherwise?

Also, @mostlikelee do we have any failure notification when one of our vuln repos fail to do its job?

[lukeheath]

@sharon-fdm Yeah, I expect there isn't any hooked up yet. MDM ran into a case where the Fleet server knew the SCEP certificate was expired, but there was no mechanism to alert beyond server logs (from my 50k' view). Since that broke MDM functionality, we would have wanted to know about it in #help-p1.

I'm wondering if EO has anything like that, where an error that would otherwise be logged to the server is something that we'd actually like to know about ASAP in #help-p1. There may not be anything like that for EO since y'all don't deal with as many certs as MDM.

[getvictor]

@sharon-fdm Normal GitHub actions workflows can do a Slack notification on fail, like:

fleet/.github/workflows/test-go.yaml

Line 147 in 223e1f2

- name: Slack Notification

[roperzh]

Hey folks, sorry for not being clear (I created the issue in a hurry)

I updated the title now, as Luke mentions, this is to surface cron errors that happen in Cloud, as they currently don't have proper visibility (so we can't use GitHub actions, etc)

If you have any crons that are mission critical and want to surface alerts, this is the place to ask for them :)

[mostlikelee]

Also, @mostlikelee do we have any failure notification when one of our vuln repos fail to do its job?
yes, this alerts to #help-p2

I can't think of anything critical in EO, but I think it would be a good metric to monitor cron failure counts.

[georgekarrv]

Hey team! Please add your planning poker estimate with Zenhub @dantecatalfamo @ghernandez345 @gillespi314 @mna @roperzh @jahzielv

[iansltx]

Based on @ksatter in https://fleetdm.slack.com/archives/C051QJU3D0V/p1723414521461399?thread_ts=1723413693.587369&cid=C051QJU3D0V, we need to mark jobs as something other than "completed" (e.g. "failed") when they fail, at which point we can filter on that + cron in CloudWatch Logs Insights to get something that we can trigger alerting on.

If we had had that alerting on the vulnerabilities cron (including marking as failed), I think we would've caught #21239 Friday afternoon. If that's enough reason to pull this into the sprint, happy to take it as a way to learn more about that part of the codebase.

[sgress454]

After reading the comments so far and looking at the current implementation, I have a slightly different proposal. I like the idea of a "failed" status for the cron jobs, but instead of trying to correlate a failed job with a log message, how about adding a new field to the cron_stats table (e.g. "failed_reason" or "notes")? Then we can update the existing monitoring Lambda to check for new failed jobs, and send an SNS message directly from the Lambda using the persisted failure reason. This Lambda already reads from the table and sends SNS messages so the new functionality would be in keeping with the existing use. The only gap I see is that the current setup only provides for configuring a single SNS topic which is directed to #help-p1, so if we wanted failures from certain jobs to go to #help-p2 we'd need to set up a new topic in Terraform and account for it in the Lambda code.

[rfairburn]

I like the idea of the db table recording the status. It would make processing in the lambda much easier.

We should still -also- include failures in the logs for those outside of cloud/aws not running the lambda, however.

[lukeheath]

From call with @sgress454:

• Finding that there are a lot of cron errors that are going off constantly.
• Most of these are not real errors, but things like noops that fail.
• So we need to go through and review all the cron jobs and make sure they're erroring unless they really error.

[lukeheath]

@sgress454 I merged this and put it on the orchestration board for tracking. QA is just validating the alert works before closing the issue out. Thanks!

[lukeheath]

@sgress454 @rfairburn Just confirming this has been released to dogfood. I haven't seen any job failures reported, have we had any in dogfood?

Also, will this be naturally rolled out to all managed cloud instances at the next update? Any additional steps necessary to test this out?

[sgress454]

This was on my TODO list for this afternoon. It's very possible that Dogfood doesn't have any failures at this time, but I'm going to look at the db and check manually for any rows with errors recorded.

Also, will this be naturally rolled out to all managed cloud instances at the next update? Any additional steps necessary to test this out?

In order to utilize this feature, instances will have to update to the latest monitoring plugin as we did with Dogfood in #24425. We can roll this out to managed instances as quickly or slowly as we like, but it won't happen automatically. We could start with one of the instances known to have had failures in the past.

[sgress454]

I could also probably simulate (stimulate?) a failure on Dogfood in order to test this e2e.

[rfairburn]

We definitely should simulate a failure in Dogfood in order to validate this end to end before rolling to cloud.

[rfairburn]

Once we are confident in Dogfood, I would prefer to roll this out to the entire managed cloud at once for consistency sake.

The number of unique differences per tenant are too great as-is already.

[sgress454]

We definitely should simulate a failure in Dogfood in order to validate this end to end before rolling to cloud.

Yup, the code worked but the SNS notification failed due to a permissions issue, I missed updating the IAM policies for the Lambda role.

2025/01/07 22:48:51 main.go:82: AuthorizationError: User: arn:aws:sts::160035666661:assumed-role/fleet-dogfood-cron-monitoring-lambda/fleet-dogfood_cron_monitoring is not authorized to perform: SNS:Publish on resource: arn:aws:sns:us-east-2:160035666661:fleet-dogfood-p2-alerts because no identity-based policy allows the SNS:Publish action

See #25268

[sgress454]

Realized we never circled back to this -- we had a successful test of the alert on Dogfood, so I think the only thing required to deploy this to managed cloud customers would be to update their monitoring add-on version to the latest.