Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grafanas show empty due to how variables are refreshed #3237

Closed
Tracked by #3469
yuvipanda opened this issue Oct 5, 2023 · 8 comments · Fixed by #3473
Closed
Tracked by #3469

Grafanas show empty due to how variables are refreshed #3237

yuvipanda opened this issue Oct 5, 2023 · 8 comments · Fixed by #3473
Assignees
@yuvipanda @consideRatio @GeorgianaElena
Labels
nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt

Comments

@yuvipanda
Copy link
Member

A lot of Grafana dashbaords are currently showing 'no data'

image

But the data is there, the problem is how variables are refreshed. Particularly the $hub variable. Even though on the top left it says 'All' that's not actually true.

image

Note that the dropdown doesn't actually contain any actual values.

If you go to settings -> variables -> $hub and scroll down, you'll see it's empty:

image

And selecting Run query does not help either with refreshing the list of variables.
@github-project-automation github-project-automation bot moved this to Needs Shaping / Refinement in DEPRECATED Engineering and Product Backlog Oct 5, 2023
@consideRatio
Copy link
Contributor

I ran into this ~2 minutes ago and didn't understand what was going on! Thanks for helping me understand this related to a grafana query issue!

@yuvipanda
Copy link
Member Author

yuvipanda commented Oct 5, 2023 •
edited by consideRatio
Loading

Temporary fix is:

  1. Set time range to 30 days (smaller amount doesn't seem to work)
  2. Go back into dashboard settings -> variables -> $hub
  3. HIt 'run query' at the bottom. List of hub values should show up here now.

@yuvipanda yuvipanda mentioned this issue Oct 7, 2023
Closed
@consideRatio
Copy link
Contributor

This is a hub response summary for a lot of different hubs. I think it could be relevant to increase the request from 10m as we have now (0.01 or 1%) to something a bit more, like 50m. But what we have now seems to not cause major trouble.

image

The ucmerced and temple hub are active hubs with ~100 users, they have peaks around 100m, and regular usage at ~30m.

image

Inactive hubs are typically idling at 10m.

image

@consideRatio consideRatio added the nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt label Oct 16, 2023
@consideRatio consideRatio mentioned this issue Oct 17, 2023
Closed
@yuvipanda
Copy link
Member Author

Another workaround is to put .* in the 'custom all' value for the $hub variable:

image

yuvipanda added a commit to yuvipanda/grafana-dashboards that referenced this issue Oct 19, 2023
@yuvipanda
Explicitly set allValues for hub variable
3ec1b7e 
Currently, on latest grafana, the dashboards will currently
show 'no data' because the `$hub` variable is not set correctly
(see 2i2c-org/infrastructure#3237 for
details). The longer term solution to this is:

1. Move away from the deprecated graphPanel to timeseries objects
   for panel
2. jupyterhub#89

Both of these are being undertaken in
jupyterhub#90,
but in the meantime, this PR implements
[this
workaround](jupyterhub#90)
so end users can at least see data.
@yuvipanda yuvipanda mentioned this issue Oct 19, 2023
Merged
@yuvipanda
Copy link
Member Author

I've opened jupyterhub/grafana-dashboards#91 which will implement the latest workaround for all hubs so we don't end up with 'no data'

@yuvipanda
Copy link
Member Author

This PR has been deployed on the openscapes hub and tested.

@consideRatio
Copy link
Contributor

PR merged upstream, re-deploying grafana dashboards via https://github.com/2i2c-org/infrastructure/actions/runs/6573134344

@damianavila damianavila moved this to Waiting 🕛 in Sprint Board Oct 27, 2023
@damianavila damianavila moved this from Needs Shaping / Refinement to Waiting in DEPRECATED Engineering and Product Backlog Oct 27, 2023
@consideRatio consideRatio moved this from Waiting 🕛 to In Progress ⚡ in Sprint Board Oct 30, 2023
@consideRatio consideRatio moved this from Waiting to In progress in DEPRECATED Engineering and Product Backlog Oct 30, 2023
This was referenced Nov 28, 2023
Closed
Merged
@github-project-automation github-project-automation bot moved this from In Progress ⚡ to Done 🎉 in Sprint Board Nov 30, 2023
@GeorgianaElena GeorgianaElena self-assigned this Nov 30, 2023
@yuvipanda
Copy link
Member Author

jupyterhub/grafana-dashboards#100 was a follow up PR that was needed

yuvipanda added a commit to yuvipanda/grafana-dashboards that referenced this issue Jul 25, 2024
@yuvipanda
Explicitly set allValues for hub variable
25abe3f 
Currently, on latest grafana, the dashboards will currently
show 'no data' because the `$hub` variable is not set correctly
(see 2i2c-org/infrastructure#3237 for
details). The longer term solution to this is:

1. Move away from the deprecated graphPanel to timeseries objects
   for panel
2. jupyterhub#89

Both of these are being undertaken in
jupyterhub#90,
but in the meantime, this PR implements
[this
workaround](jupyterhub#90)
so end users can at least see data.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuvipanda yuvipanda

@consideRatio consideRatio

@GeorgianaElena GeorgianaElena

Labels
nominated-to-be-resolved-during-q4-2023 Nomination to be resolved during q4 goal of reducing the technical debt
Projects
No open projects
DEPRECATED Engineering and Product Backlog
Status: Complete
Sprint Board
Status: Done 🎉
Milestone
No milestone
3 participants
@yuvipanda @consideRatio @GeorgianaElena