Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update queries for workflow classification counts - attempt to optimize workflow classification count performance #70

Closed
wants to merge 20 commits into from

Conversation

yuenmichelle1
Copy link
Collaborator

Because of how the FE calls out /classifications?workflow_id endpoint when querying from project's stats (i.e. all at once), which is hammering the API and Timescale db. 🔨

This ^ (and our non-usage of compression or data retention policies) plus our usage of Real Time Aggregates on the continuous aggregate we query workflow data from, causes querying to be not performant. (As evident when turning off Real Time Aggregation, we see faster query times and lower cpu utilization).

Because we are limited in resources and we still want Real Time Aggregation to be possible, we do the following for querying workflow classification counts ONLY:

  • Create a New HourlyClassificationCountByWorkflow RealTime Aggregate
  • Set a data retention policy for this new hourly workflow classification count aggregate (this should limit the amount of data the query planner will have to sift through when dealing with real time aggregates)
  • Turn off real time aggregation for the DailyClassificationCount
  • When querying workflow classification counts that include current date's counts, we query current date's counts via the HourlyClassificationCountByWorkflow continuous aggregate and we query the materialized DailyClassificationCountsByWorkflow for everything before that date

app/queries/count_classifications.rb Show resolved Hide resolved
app/queries/count_classifications.rb Show resolved Hide resolved
app/queries/count_classifications.rb Show resolved Hide resolved
app/queries/count_classifications.rb Show resolved Hide resolved
db/schema.rb Show resolved Hide resolved
@@ -0,0 +1,27 @@
# frozen_string_literal: true.
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Style/FrozenStringLiteralComment: Missing magic comment # frozen_string_literal: true.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant