Katib UI experiment details slow query

[a9p]

/kind bug

What steps did you take and what happened:

Loading the Experiment Details view for an experiment that has more than a few hundred trials results in complete page load latency on the order of minutes.

What did you expect to happen:

Experiment Details view loads quickly, ideally with an upper bound on the order of seconds.

Anything else you would like to add:

• Originating discussion
• A possible culprit looks to be the fetch_hp_job_info endpoint

Environment:

• Katib version (check the Katib controller image version): 0.15.0
• Kubernetes version: (kubectl version): 1.24.x
• OS (uname -a): ubuntu 22

---

Impacted by this bug? Give it a 👍 We prioritize the issues with the most 👍

[a9p]

Related issues:

• Experiment stuck due to hitting Suggestion custom resource size limits #1847
• Persist meta data of experiments in DB #1972

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[andreyvelich]

/lifecycle frozen
/help

[google-oss-prow]

@andreyvelich:
This request has been marked as needing help from a contributor.

Please ensure the request meets the requirements listed here.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-help command.

In response to this:

/lifecycle frozen
/help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.