Enable DB profiling for edxapp

[robrap]

AC:

• Enable DB profiling for the edxapp DB.

• This may need to be an SRE ticket. We could use the escalate label.

• See https://docs.datadoghq.com/database_monitoring/setup_mysql/aurora/?tab=mysql56

• Blocked: See Revert "chore: Enable Data streaming and APM profiling edxapp, forum and django service" configuration#30 for revert of the last time we tried this. This work is blocked by NR being removed from edxapp, because we think that was the problem.

[robrap]

Where did this feature go in DD, and can we see an example of it in DD to know whether it would be useful before enabling? We would just enable it if it didn't have any risk.

[robrap]

[inform] I'm marking this as blocked for a bit. I'm hoping we can resolve the NR issue, and unblock this work with less risk. Also, Jeremy is on vacation and I want to minimize risk during that time.

[dianakhuang]

Created an internal SRE ticket to enable this: https://2u-internal.atlassian.net/browse/DOS-5330

[jristau1984]

Adding a due date to check back in on the linked DOS ticket again.

If we go the AWS Cloudwatch route for edxapp, can we possibly use DB profiling for other services to take advantage of this paid DD feature?

[robrap]

@jristau1984: [inform] I switched from "Due Date" to "Review After Date".

[robrap]

Nadeem responded, and we should review: https://2u-internal.atlassian.net/browse/DOS-5330. Looks like a db reboot is required. How does that affect the plan?

[jristau1984]

Yikes, that requires planned downtime correct? I assume we do not have HA for our DBs...

I think, if this is a valuable enough feature, that we can schedule a planned downtime for it. However, we should be very confident that we will NOT need to flip it back off and on to figure out if it will have an impact on performance.

[robrap]

@jristau1984: I wrote my version of the open questions on the DOS ticket. We'll see how Nadeem responds.

[robrap]

@jristau1984: Let me know if you are watching the DOS ticket, and if I don't need to report everything here.

That said, the reboot will cause a small interruption during failover, and not an extended downtime. This was not clear before.

I gave the go ahead for an Edge deployment. Then we'll move to Prod. I asked if SRE has a process for planning and communicating these types of changes, but if you (or Spencer) wish to be involved in that, please just insert yourself. Thank you.

[robrap]

@dianakhuang: I took this out of blocked and moved it to in-progress. Would you be able to follow-up on the SRE ticket again, now that its status has changed?