workflow: move build operations in a reusable workflow #244
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
chantra
force-pushed
the
reuseable_workflow_tests
branch
4 times, most recently
from
e08d567 to
b6206d7
Compare
chantra
force-pushed
the
reuseable_workflow_tests
branch
22 times, most recently
from
8ff1d21 to
bb192fa
Compare
This is step 1 of running the kernel build from a re-useable workflow. This change simply move the "build" job in its own reusable workflow file. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Signed-off-by: Manu Bretelle <chantr4@gmail.com>
…h/toolchain matrix.py was modified to embed the tests in the build-matrics, this way we can trigger all tests within 1 workflow without needing to do some filtering in the GH action side. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
chantra
force-pushed
the
reuseable_workflow_tests
branch
from
bb192fa to
1ac463c
Compare
Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Signed-off-by: Manu Bretelle <chantr4@gmail.com>
|
I have ran KPD against this and it seems that it will handle updating KPD just fine. I am also opening this PR against a different branch than master so we can switch KPD to use the new vmtest code and possibly revert back to main which we know is stable. |
yurinnick
reviewed
Oct 26, 2023
This makes it nicer in the UI vs the auto-generated name... which is super long because they include the matrix parameters as part of it and it ended looking like: build-and-test (LATEST, self-hosted, x86_64, x86_64, llvm, 16, llvm-16, test_progs, false, 360, t... / test (test_progs, false, 360) / test_progs on x86_64 with llvm-16 Signed-off-by: Manu Bretelle <chantr4@gmail.com>
At this stage, we are already not using `test-matrix` anymore, so let's remove it
and move the logic of adding the tests within the first loop over the matrix.
For where to run veristat, as eluded to in a comment I am removing, we can control
that from within matrix.py directly.
Finally, we probably don't care on what/where veristat runs on. Let's default
to wherever we are currently running builds and tests.
Unless specified on the matrix, we will not run veristat.
Finally, here is what the output of matrix looks like
```
GITHUB_REPOSITORY_OWNER=kernel-patches GITHUB_REPOSITORY=vmtest GITHUB_OUTPUT=/dev/stdout python3 .github/scripts/matrix.py | cut -d= -f2- | jq .
{
"include": [
{
"kernel": "LATEST",
"runs_on": [
"ubuntu-20.04"
],
"arch": "x86_64",
"toolchain": "gcc",
"llvm-version": "16",
"run_veristat": true,
"toolchain_full": "gcc",
"tests": {
"include": [
{
"test": "test_progs",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_progs_parallel",
"continue_on_error": true,
"timeout_minutes": 30
},
{
"test": "test_progs_no_alu32",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_progs_no_alu32_parallel",
"continue_on_error": true,
"timeout_minutes": 30
},
{
"test": "test_maps",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_verifier",
"continue_on_error": false,
"timeout_minutes": 360
}
]
}
},
{
"kernel": "LATEST",
"runs_on": [
"ubuntu-20.04"
],
"arch": "x86_64",
"toolchain": "llvm",
"llvm-version": "16",
"toolchain_full": "llvm-16",
"run_veristat": false,
"tests": {
"include": [
{
"test": "test_progs",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_progs_parallel",
"continue_on_error": true,
"timeout_minutes": 30
},
{
"test": "test_progs_no_alu32",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_progs_no_alu32_parallel",
"continue_on_error": true,
"timeout_minutes": 30
},
{
"test": "test_maps",
"continue_on_error": false,
"timeout_minutes": 360
},
{
"test": "test_verifier",
"continue_on_error": false,
"timeout_minutes": 360
}
]
}
}
]
}
```
Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Add a new job to run matrix.py with a combination of owner/repo to make sure we don't raise. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Add a step to run black on our python scripts. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
chantra
force-pushed
the
reuseable_workflow_tests
branch
from
2695dda to
6831f53
Compare
danielocfb
approved these changes
Oct 26, 2023
| # builds (which may check out some earlier upstream change). | ||
| with: | ||
| fetch-depth: 50 | ||
| - if: ${{ github.repository == 'kernel-patches/vmtest' }} |
There was a problem hiding this comment.
I'd rather we make this some kind of meaningful input argument than this reliance on global state. Do you think that's possible?
chantra
commented
Oct 26, 2023
chantra
added a commit
to chantra/kernel-patches-vmtest
that referenced
this pull request
Oct 27, 2023
@danielocfb brought this up in kernel-patches#244. e.g we should control whether to execute the test job not from the job itself, but from the caller. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
chantra
added a commit
that referenced
this pull request
Oct 27, 2023
@danielocfb brought this up in #244. e.g we should control whether to execute the test job not from the job itself, but from the caller. Signed-off-by: Manu Bretelle <chantr4@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This change essentially isolated the old jobs:
in re-usable workflows:
.github/workflows/kernel-{build,test,veristat}.yml.There is then another reusable workflow:
.github/workflows/kernel-build-test.ymlwhich runs the jobs to build, test, and run veristat for a specific architecture/toolchain.Finally,
.github/workflows/test.ymlis still the main workflow, but now run theset-matrixjob and initiate akernel-build-testworkflow for each arch/toolchain pairs.This allow us to essentially start those workflow in parallel, as a build in a specific workflow is finished, it will process with testing directly without waiting for other jobs.
Finally, matrix.py was changed to include the tests to run directly as part of the
build-matrix. This allow us to not have to do any filtering as part of the GH actions.In the future, the test-matrix could be removed. In the future, we may also want to embed whether or not we run veristat at the build matrix level.... which is not a build-matrix anymore, but an actual full end-to-end matrix.