Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

progress indicator #223

Open
BenTheElder opened this issue Dec 16, 2024 · 3 comments
Open

progress indicator #223

BenTheElder opened this issue Dec 16, 2024 · 3 comments
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/testing Categorizes an issue or PR as relevant to SIG Testing.

Comments

@BenTheElder
Copy link
Member

It would be nice to have some idea how far along the running tests has gotten, given the multi-hour duration of conformance tests.
/kind feature
/sig testing

Right now there's nothing logged until the test completes, and the spinner doesn't inspire confidence over multiple hours that we're actually getting anywhere.

cc @pohly for thoughts on how best to integrate with the ginkgo reporter.

My rough thinking: We should be able to have ginkgo at least print a "." per test and then count those and include them alongside the spinner, plus parse the "will run x out of y specs"

Something like:

$ hydrophone --conformance
22:16:26 INF API endpoint: https://127.0.0.1:36061
22:16:26 INF Server version: version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.0", GitCommit:"70d3cc986aa8221cd1dfb1121852688902d3bf53", GitTreeState:"clean", BuildDate:"2024-12-13T19:19:34Z", GoVersion:"go1.23.3", Compiler:"gc", Platform:"linux/amd64"}
22:16:26 INF Using namespace: conformance
22:16:26 INF Using conformance image: registry.k8s.io/conformance:v1.32.0
22:16:26 INF Using busybox image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
22:16:26 INF Test framework will start 1 thread(s) and use verbosity level 4.
22:16:26 INF Created Namespace conformance.
22:16:26 INF Created ServiceAccount conformance-serviceaccount.
22:16:26 INF Created Clusterrole conformance-serviceaccount:conformance.
22:16:26 INF Created ClusterRoleBinding conformance-serviceaccount-role:conformance.
22:16:26 INF Created Pod e2e-conformance-test.
22:16:26 INF Waiting up to 5m0s for Pod to start...
⠈⠁2024/12/16 22:16:29 Running command:
Command env: []
Run from directory: 
Executable path: /usr/local/bin/ginkgo
Args (comma-delimited): /usr/local/bin/ginkgo,--focus=\[Conformance\],--skip=,--no-color=true,--timeout=24h,/usr/local/bin/e2e.test,--,--disable-log-dump,--repo-root=/kubernetes,--provider=skeleton,--report-dir=/tmp/results,--kubeconfig=
2024/12/16 22:16:29 Now listening for interrupts
  I1216 22:16:30.364816      33 e2e.go:109] Starting e2e run "5e702b17-ac49-47b3-b5ac-fdede8df27dd" on Ginkgo node 1
Running Suite: Kubernetes e2e suite - /usr/local/bin
====================================================
Random Seed: 1734387389 - will randomize all specs

Will run 411 of 6622 specs
⠈⡱

Could instead be like:

$ hydrophone --conformance
22:16:26 INF API endpoint: https://127.0.0.1:36061
22:16:26 INF Server version: version.Info{Major:"1", Minor:"32", GitVersion:"v1.32.0", GitCommit:"70d3cc986aa8221cd1dfb1121852688902d3bf53", GitTreeState:"clean", BuildDate:"2024-12-13T19:19:34Z", GoVersion:"go1.23.3", Compiler:"gc", Platform:"linux/amd64"}
22:16:26 INF Using namespace: conformance
22:16:26 INF Using conformance image: registry.k8s.io/conformance:v1.32.0
22:16:26 INF Using busybox image: registry.k8s.io/e2e-test-images/busybox:1.36.1-1
22:16:26 INF Test framework will start 1 thread(s) and use verbosity level 4.
22:16:26 INF Created Namespace conformance.
22:16:26 INF Created ServiceAccount conformance-serviceaccount.
22:16:26 INF Created Clusterrole conformance-serviceaccount:conformance.
22:16:26 INF Created ClusterRoleBinding conformance-serviceaccount-role:conformance.
22:16:26 INF Created Pod e2e-conformance-test.
22:16:26 INF Waiting up to 5m0s for Pod to start...
2024/12/16 22:16:29 Running command:
Command env: []
Run from directory: 
Executable path: /usr/local/bin/ginkgo
Args (comma-delimited): /usr/local/bin/ginkgo,--focus=\[Conformance\],--skip=,--no-color=true,--timeout=24h,/usr/local/bin/e2e.test,--,--disable-log-dump,--repo-root=/kubernetes,--provider=skeleton,--report-dir=/tmp/results,--kubeconfig=
2024/12/16 22:16:29 Now listening for interrupts
2024/12/16 22:16:30 Tests started with 411 out of 6622 to run

⢎⡠ [200/411] ...
@k8s-ci-robot k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. sig/testing Categorizes an issue or PR as relevant to SIG Testing. labels Dec 16, 2024
@BenTheElder
Copy link
Member Author

(Or even no spinner but just periodically print a count of completed tests / total tests with the current timestamp)

We used to have an experience like this when manually running the conformance e2e tests against tools / products / ... for submitting them to https://github.com/cncf/k8s-conformance, this feels like a regression between the assorted tooling.

@BenTheElder
Copy link
Member Author

I see that currently we don't have the pod configured to print anything:

$ kubectl logs -n=conformance e2e-conformance-test
Defaulted container "conformance-container" out of: conformance-container, output-container
2024/12/16 22:16:29 Running command:
Command env: []
Run from directory: 
Executable path: /usr/local/bin/ginkgo
Args (comma-delimited): /usr/local/bin/ginkgo,--focus=\[Conformance\],--skip=,--no-color=true,--timeout=24h,/usr/local/bin/e2e.test,--,--disable-log-dump,--repo-root=/kubernetes,--provider=skeleton,--report-dir=/tmp/results,--kubeconfig=
2024/12/16 22:16:29 Now listening for interrupts
  I1216 22:16:30.364816      33 e2e.go:109] Starting e2e run "5e702b17-ac49-47b3-b5ac-fdede8df27dd" on Ginkgo node 1
Running Suite: Kubernetes e2e suite - /usr/local/bin
====================================================
Random Seed: 1734387389 - will randomize all specs

Will run 411 of 6622 specs

But we can fix that.

@BenTheElder
Copy link
Member Author

BenTheElder commented Dec 16, 2024

We're actually already logging like this:

SSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSS•SS•SSSSS•SSSSSSSSSSSSSSSSSSSSSS•SS•SSS•SSSSSSSSSSSSSSS•SSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SSSSSSS•SSSSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSS•SS•SSSSSSSSSSSSSSSS•SSS•SSS•SSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSSSSS•SSSSSSSSSSSSSSS•SSSSS•SSSSSSSSSSSSSS•S•SSSSSSSSSS•SSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSS•SSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SSSSSSSSSSSSSSSS••SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS••SSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSSSSS••S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSS•SSSSS••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSS•SSSSSSSSSSS•S•SSSSSSSSSSSS••SSSSSSSSSSSSSS•SSSSSSS•SSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSS•SSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSS••SS•SSSSSSS•SSSSSSSSSS••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSS•SSSSSSS•SSSSSS•S•SSSSSSSSSSSSSSS•SSSSS•SSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSS•SSSSSSSS••SSSSS•SSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSS•SSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSS•SSSSSSSSS•SSSSS•SSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSS•SSSSSSS•S•SSSSSSS•SSSSSSSSSSSSSSSSSSSS•SSSS•SS••SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSS•SSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SS•SSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SSSSSSSSSSS••SSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSS•SSSSSSS•SSSS•SS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSS•SSS•SSSSSSSSSSS•SSSSSSSSSSSSS•SSSS•SSSS•SSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSS•SSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SS•S•SSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSS••SSSSSSSSSS•SSSSSS•SSS•SSSSS••S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSS•SSSSSS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSS•SSSSSSSSSSSSS•SSSSS•SSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SS•SSSSSSSSSSS•S•SSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSS••SSSS•SSSSSSSSSSS•SSSSSSSSSSSS•S••S•SSSSSSSSSSSSS•SS••SSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSS•SSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSS•SSSSSSSSSSSSSSS•SSSSSSSSS•SSSSS•SSS•SSSSSSSSSSSSSSSSSSS••SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSS•SSS•SSSSSSSSSSSSSSS•SS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSS•SS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSS•SSSSSSSSSSSS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSS•SSSSSSSSSSSSS•SS•SSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSS•SSSSSSSSSS•S•SSSS•SSSSSSSSS•SSSSSSSSSSSS•SSS•SSSSSSSSS•SSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSS•SSSSSSS•SSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSSSSS•S•SSSSSSSS•S•SSSSSSSSSSSSSSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•S•SSSSSSSSSSSSSSSS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS••SSSS•SSSSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSS•SSS•SSSSSSSSSS•SSSSSSSSSS•SSSSSSS•SSSSSSSSSSSSSSS•SSS•SSSSSSSSSSSSSSSSSSSSSSSS•SSSSS••SS•S•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSSSSSSSS•SSSSSSSSSSSSSSSSSSS•S•S•S•SSSSSSS

Just only in /tmp/results/e2e.log, and not to the output of hydrophone.

So we can get the total from the Will run 411 of 6622 specs line and then get progress like:
kubectl exec -n=conformance e2e-conformance-test -c output-container -- cat /tmp/results/e2e.log | grep -o "•" | wc -l
(except written in go)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature Categorizes issue or PR as related to a new feature. sig/testing Categorizes an issue or PR as relevant to SIG Testing.
Projects
None yet
Development

No branches or pull requests

2 participants