Improve CI compile times by fixing the sccache mechanism #2208
Conversation
Try to setup s3 sccache Add SCCACHE environment variables to build workflow Update SCCACHE configuration in build workflow to include region and SSL settings Remove unnecessary GitHub default packages in build workflow to save space Update condition for enabling Windows compiler commands in build workflow Simplify conditions for installing MSVC and enabling Windows compiler commands in build workflow Remove /Zm flags from compiler options and restore PCH usage in CMake presets Remove /Zm flags from CMakeLists and disable PCH usage in CMake presets Clarify SCCACHE_GHA_VERSION environment variable comment in build_steps.yml
G-D-Petrov
requested review from
alexowens90,
willdealtry and
poodlewars
as code owners
| @@ -101,16 +110,12 @@ jobs: | |||
| CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}} | |||
|
|
|||
| # ========================= Leader steps ========================= | |||
| - name: Remove GitHub default packages (baseimage) # To save space | |||
There was a problem hiding this comment.
This is no longer needed as the cache is reducing the compilation load, i.e. no as much disk space is needed.
There was a problem hiding this comment.
May still need it if we ever tear down (or invalidate) the whole cache?
There was a problem hiding this comment.
Doesn't seem like it, the wheels build was able to run even when an empty cache (when I changed to S3)
The step below is still needed for C++ tests jobs.
G-D-Petrov
added
bug
.github/workflows/build_steps.yml
Outdated
| @@ -48,7 +48,15 @@ jobs: | |||
| runs-on: ${{ needs.start_ec2_runner.status != 'failure' && needs.start_ec2_runner.outputs.label || matrix.distro}} | |||
| container: ${{ (matrix.os == 'linux' && inputs.job_type != 'build-python-wheels') && matrix.container || null}} | |||
| env: | |||
| SCCACHE_GHA_VERSION: ${{vars.SCCACHE_GHA_VERSION || 1}} # Setting this env var enables the caching | |||
| # 0 - uses S3 Cache, 1 - does uses GHA cache | |||
There was a problem hiding this comment.
typo does uses should be uses
.github/workflows/build_steps.yml
Outdated
| @@ -48,7 +48,15 @@ jobs: | |||
| runs-on: ${{ needs.start_ec2_runner.status != 'failure' && needs.start_ec2_runner.outputs.label || matrix.distro}} | |||
| container: ${{ (matrix.os == 'linux' && inputs.job_type != 'build-python-wheels') && matrix.container || null}} | |||
| env: | |||
| SCCACHE_GHA_VERSION: ${{vars.SCCACHE_GHA_VERSION || 1}} # Setting this env var enables the caching | |||
| # 0 - uses S3 Cache, 1 - does uses GHA cache | |||
| # this was extract PRs can use the GHA cache | |||
There was a problem hiding this comment.
Think there's a typo in this sentence, I don't understand it
| # this was extract PRs can use the GHA cache | ||
| SCCACHE_GHA_VERSION: ${{secrets.AWS_S3_ACCESS_KEY && 0 || 1}} | ||
| SCCACHE_BUCKET: arcticdb-ci-sccache-bucket | ||
| SCCACHE_ENDPOINT: http://s3.eu-west-1.amazonaws.com |
There was a problem hiding this comment.
Do you know what region our CI runners run in? Allegedly they're all in the US, so that might be a better location for the cache.
There was a problem hiding this comment.
I will test this next week
| @@ -101,16 +110,12 @@ jobs: | |||
| CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}} | |||
|
|
|||
| # ========================= Leader steps ========================= | |||
| - name: Remove GitHub default packages (baseimage) # To save space | |||
There was a problem hiding this comment.
May still need it if we ever tear down (or invalidate) the whole cache?
Reference Issues/PRs
Monday ref: 8590725039
What does this implement or fix?
Currently, the sscache mechanism doesn't work properly because:
This PR changes the flows to:
This results in 3-4x speed up for compilation (10-15 min now vs 30-45 min before) for all builds.
Now the Linux builds + tests take less than an hour ( vs ~90 min before)
Any other comments?
Checklist
Checklist for code changes...