API response time assessment and reduction

[AetherUnbound]

Description

This project thread is intended to capture the work related to investigating and reducing our APIs average response time. Recent elevated response times have impeded our ability to run image data refreshes, so we'd like to understand the response times more and reduce the effect the data refresh has on our API's average response time.

Issues

Project board for managing this work is here: https://github.com/orgs/WordPress/projects/144

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

@dhruvkb and @obulat, with the assistance of @stacimc and @sarayourfriend were able to deploy the index reduction changes outlined in the deployment plan and reduce the size of the indices by 30%! This is a tremendous accomplishment, but it unfortunately did not seem to affect the API response time as we had hoped.

We are continuing to work on increased monitoring for Elasticsearch, and other tickets related to improving the data refresh process so we can unpause it once more.

[openverse-bot]

Hi @AetherUnbound, this project has not received an update comment in 14 days. Please leave an update comment as soon as you can. See the documentation on project updates for more information.

[AetherUnbound]

We recently deployed three major changes which we believe have helped address the issues:

• Blocked some potentially malicious users of the API
• Improved how the /related/ queries are run (via Simplify the related search #3151)
• Changed the API to ASGI (via Convert the API to ASGI #2790)

This has had a tremendous effect on the /related/ endpoints and reduced the periodicity of these spikes!

While we still have a few issue left in the project view, work on other projects can resume now that the response times have stabilized.

We're also in the process of restarting the data refreshes which have been paused while we address this. Audio has completed already, and we will start image later this week.

[AetherUnbound]

After a few more days of monitoring, the difference in Elasticsearch (and thus API) response time is evident!

We believe this to be a product of the 3 mentioned changes above. We've moved this project into "Shipped" for now until we get some feedback around the simple query string changes (#3327) and deploy the reduced data node JVM heap change (https://github.com/WordPress/openverse-infrastructure/issues/677, currently waiting on the data refresh). We've been able to run data refreshes successfully without additional strain on the API, I think we can safely say we're in the clear here! 🥳

[sarayourfriend]

Closing this issue as we've resolved the API response time problems.