kfp-api from channel 2.0-alpha.7/stable gets stuck in Waiting for relational-db data

[orfeas-k]

Bug Description

During deployment of CKF bundle from bundle file 1.7/stable, I noticed that deployment was not successful with due to kfp-api being stuck in Waiting state with message Waiting for relational-db data. However, kfp-db was Active since early stages of the bundle's deployment. Looking into the logs, I see

unit-kfp-api-0: 18:00:22 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 18:00:22 ERROR unit.kfp-api/0.juju-log Failed to handle <UpdateStatusEvent via KfpApiOperator/on/update_status[251]> with error: Waiting for relational-db data

To Reproduce

Run the Deploy bundle and run UATs workflow with tests-bundle/1.7 as bundle tests and --file=releases/1.7/stable/kubeflow/bundle.yaml as bundle source. Note that the issue is intermittent and doesn't take place on every run of the workflow.

Environment

Self hosted runners, Microk8s 1.24, juju 2.9

Relevant log output

unit-kfp-api-0: 18:00:22 ERROR unit.kfp-api/0.juju-log Failed to generate container configuration.
unit-kfp-api-0: 18:00:22 ERROR unit.kfp-api/0.juju-log Failed to handle <UpdateStatusEvent via KfpApiOperator/on/update_status[251]> with error: Waiting for relational-db data

Additional context

No response