FIX secrets and mysql app selector for kubeflow install #2510

pedrovgp · 2025-02-06T16:48:09Z

This is the original PR (with linked issue) that originated these changes.

FOR REVIEWERS: the first commit is the relevant one, the second is fixing deprecation warnings from patcheStrategicMerge and vars.

google-oss-prow · 2025-02-06T16:48:14Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tenzen-y for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Pedro Viana <pedro.viana@nubank.com.br>

juliusvonkohout · 2025-02-07T14:11:01Z

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml

@@ -11,18 +11,16 @@ resources:
 images:
  - name: docker.io/kubeflowkatib/katib-controller
    newName: docker.io/kubeflowkatib/katib-controller
-    newTag: latest
+    newTag: v0.17.0


Is latest not the correct tag on the master branch?

tenzen-y

As I checked your commit in my local, this seems not to be available:

Error from server (InternalError): error when creating "examples/v1beta1/hp-tuning/grid.yaml": Internal error occurred: failed calling webhook "defaulter.experiment.katib.kubeflow.org": failed to call webhook: Post "https://katib-controller.kubeflow.svc:443/mutate-experiment?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

{"level":"info","ts":"2025-02-08T15:12:33Z","msg":"Starting workers","controller":"suggestion-controller","worker count":1}
{"level":"info","ts":"2025-02-08T15:12:33Z","msg":"Starting workers","controller":"experiment-controller","worker count":1}
2025/02/08 15:14:41 http: TLS handshake error from 10.244.0.1:63983: remote error: tls: bad certificate
2025/02/08 15:14:44 http: TLS handshake error from 10.244.0.1:63924: remote error: tls: bad certificate
2025/02/08 15:15:28 http: TLS handshake error from 10.244.0.1:56985: remote error: tls: bad certificate
2025/02/08 15:17:44 http: TLS handshake error from 10.244.0.1:14775: remote error: tls: bad certificate

Could you focus only on 36e18c7 in this PR due to release cycle?
As separately, we can open dedicated PR only for 8cc3427.

tenzen-y · 2025-02-08T15:13:22Z

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml

@@ -11,18 +11,16 @@ resources:
 images:
  - name: docker.io/kubeflowkatib/katib-controller
    newName: docker.io/kubeflowkatib/katib-controller
-    newTag: latest
+    newTag: v0.17.0


Suggested change

newTag: v0.17.0

newTag: latest

The head branch should use the latest image.

tenzen-y · 2025-02-08T15:13:31Z

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml

  - name: docker.io/kubeflowkatib/katib-db-manager
    newName: docker.io/kubeflowkatib/katib-db-manager
-    newTag: latest
+    newTag: v0.17.0


Suggested change

newTag: v0.17.0

newTag: latest

tenzen-y · 2025-02-08T15:13:45Z

manifests/v1beta1/installs/katib-with-kubeflow/kustomization.yaml

-
-patchesStrategicMerge:
-  - patches/remove-namespace.yaml
+    newTag: v0.17.0


Suggested change

newTag: v0.17.0

newTag: latest

juliusvonkohout · 2025-02-08T15:59:44Z

As I checked your commit in my local, this seems not to be available:

Error from server (InternalError): error when creating "examples/v1beta1/hp-tuning/grid.yaml": Internal error occurred: failed calling webhook "defaulter.experiment.katib.kubeflow.org": failed to call webhook: Post "https://katib-controller.kubeflow.svc:443/mutate-experiment?timeout=10s": tls: failed to verify certificate: x509: certificate signed by unknown authority

{"level":"info","ts":"2025-02-08T15:12:33Z","msg":"Starting workers","controller":"suggestion-controller","worker count":1}
{"level":"info","ts":"2025-02-08T15:12:33Z","msg":"Starting workers","controller":"experiment-controller","worker count":1}
2025/02/08 15:14:41 http: TLS handshake error from 10.244.0.1:63983: remote error: tls: bad certificate
2025/02/08 15:14:44 http: TLS handshake error from 10.244.0.1:63924: remote error: tls: bad certificate
2025/02/08 15:15:28 http: TLS handshake error from 10.244.0.1:56985: remote error: tls: bad certificate
2025/02/08 15:17:44 http: TLS handshake error from 10.244.0.1:14775: remote error: tls: bad certificate

Could you focus only on 36e18c7 in this PR due to release cycle?
As separately, we can open dedicated PR only for 8cc3427.

Maybe you are missing cert-manager from kubeflow/manifests https://github.com/kubeflow/manifests/blob/59d1312e3f0b818607ff1f83da0574a9f6a75569/example/kustomization.yaml#L73 . This is for the Multi-User "kubeflow" installation of katib https://github.com/kubeflow/manifests/blob/59d1312e3f0b818607ff1f83da0574a9f6a75569/example/kustomization.yaml#L69

juliusvonkohout · 2025-02-08T16:02:41Z

@pedrovgp the test in /katib and /manifests are different. Please keep the same changes (PR) against kubeflow/manifest open to get feedback from our tests in kubeflow/manifests. We need to have the tests from both repositories.

andreyvelich

Thank you for creating this @pedrovgp.

katib-db-manager was using the secret katib-mysql-secret to connect to the database. But in this install (1) the database used by katib is mysql (not katib-mysql, which is not even deployed) and (2) the secrets (username and password) are different
katib-db-manager points to the Service katib-mysql (instead of pointing to the mysql service), which is fine, since the service should point to the same Deployment as the mysql Service. But the selector was not working properly, and it was not pointing to the right Deployment.

Regarding these comments, please can you explain how did it work before ?
We didn't introduce any changes in Katib MySQL and Katib DB Manager names in the last 3 releases.

pedrovgp · 2025-02-10T16:13:23Z

Hi @andreyvelich, as in my closing comment here, I could not reproduce the problem with Kind. And even in the cluster I was testing, the problem did not happen again. I believe the deployment of katib-mysql failed a previous attempt, and that led to the investigation I mentioned.

And, double checking, the service was pointing correctly to katib-mysql in my attempt to reproduce the issue. So I'll close this one.

juliusvonkohout · 2025-02-10T16:46:48Z

@pedrovgp but we still need the kustomize 5 fixes

juliusvonkohout · 2025-02-10T16:53:57Z

/reopen

google-oss-prow · 2025-02-10T16:54:02Z

@juliusvonkohout: Reopened this PR.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

juliusvonkohout · 2025-02-10T16:57:24Z

@andreyvelich @tenzen-y I just copied over the commit for now into #2511

/close

google-oss-prow · 2025-02-10T16:57:29Z

@juliusvonkohout: Closed this PR.

In response to this:

@andreyvelich @tenzen-y I just copied over the commit for now into #2511

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

google-oss-prow bot requested review from anencore94 and Electronic-Waste February 6, 2025 16:48

FIX secrets and mysql app selector for kubeflow install

36e18c7

Signed-off-by: Pedro Viana <pedro.viana@nubank.com.br>

google-oss-prow bot added the size/M label Feb 6, 2025

pedrovgp force-pushed the issue-2980-katib-db-manager branch from 0d2d278 to 36e18c7 Compare February 6, 2025 16:48

google-oss-prow bot added size/L and removed size/M labels Feb 6, 2025

pedrovgp force-pushed the issue-2980-katib-db-manager branch from b496f43 to 7c9a3c7 Compare February 6, 2025 18:36

FIX deprecation warnings

8cc3427

Signed-off-by: Pedro Viana <pedro.viana@nubank.com.br>

pedrovgp force-pushed the issue-2980-katib-db-manager branch from 7c9a3c7 to 8cc3427 Compare February 6, 2025 18:36

pedrovgp mentioned this pull request Feb 6, 2025

FIX katib-db-manager secrets and service to connect to mysql kubeflow/manifests#2981

Closed

juliusvonkohout reviewed Feb 7, 2025

View reviewed changes

This was referenced Feb 7, 2025

[Release] Katib 0.18 Roadmap #2386

Open

Adding tests for katib mysql kubeflow/manifests#2983

Closed

tenzen-y reviewed Feb 8, 2025

View reviewed changes

juliusvonkohout mentioned this pull request Feb 10, 2025

Katib upstream manifest fixes kubeflow/manifests#2985

Open

7 tasks

andreyvelich reviewed Feb 10, 2025

View reviewed changes

pedrovgp closed this Feb 10, 2025

google-oss-prow bot reopened this Feb 10, 2025

google-oss-prow bot closed this Feb 10, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

FIX secrets and mysql app selector for kubeflow install #2510

FIX secrets and mysql app selector for kubeflow install #2510

pedrovgp commented Feb 6, 2025 •

edited

Loading

google-oss-prow bot commented Feb 6, 2025

juliusvonkohout Feb 7, 2025 •

edited

Loading

tenzen-y left a comment •

edited

Loading

tenzen-y Feb 8, 2025

tenzen-y Feb 8, 2025

tenzen-y Feb 8, 2025

juliusvonkohout commented Feb 8, 2025 •

edited

Loading

juliusvonkohout commented Feb 8, 2025

andreyvelich left a comment

pedrovgp commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

google-oss-prow bot commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

google-oss-prow bot commented Feb 10, 2025

FIX secrets and mysql app selector for kubeflow install #2510

FIX secrets and mysql app selector for kubeflow install #2510

Conversation

pedrovgp commented Feb 6, 2025 • edited Loading

google-oss-prow bot commented Feb 6, 2025

juliusvonkohout Feb 7, 2025 • edited Loading

Choose a reason for hiding this comment

tenzen-y left a comment • edited Loading

Choose a reason for hiding this comment

tenzen-y Feb 8, 2025

Choose a reason for hiding this comment

tenzen-y Feb 8, 2025

Choose a reason for hiding this comment

tenzen-y Feb 8, 2025

Choose a reason for hiding this comment

juliusvonkohout commented Feb 8, 2025 • edited Loading

juliusvonkohout commented Feb 8, 2025

andreyvelich left a comment

Choose a reason for hiding this comment

pedrovgp commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

google-oss-prow bot commented Feb 10, 2025

juliusvonkohout commented Feb 10, 2025

google-oss-prow bot commented Feb 10, 2025

pedrovgp commented Feb 6, 2025 •

edited

Loading

juliusvonkohout Feb 7, 2025 •

edited

Loading

tenzen-y left a comment •

edited

Loading

juliusvonkohout commented Feb 8, 2025 •

edited

Loading