Skip to content

Improve CI compile times by fixing the sccache mechanism #2208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 2 commits into from
Closed

Improve CI compile times by fixing the sccache mechanism #2208

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4fba035
Test CI compile time improvements
G-D-Petrov Feb 26, 2025
6297c2c
Fix typos
G-D-Petrov Mar 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions .github/workflows/build_steps.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,15 @@ jobs:
runs-on: ${{ needs.start_ec2_runner.status != 'failure' && needs.start_ec2_runner.outputs.label || matrix.distro}}
container: ${{ (matrix.os == 'linux' && inputs.job_type != 'build-python-wheels') && matrix.container || null}}
env:
SCCACHE_GHA_VERSION: ${{vars.SCCACHE_GHA_VERSION || 1}} # Setting this env var enables the caching
# 0 - uses S3 Cache, 1 - does uses GHA cache
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo does uses should be uses

# this was extract PRs can use the GHA cache
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think there's a typo in this sentence, I don't understand it

SCCACHE_GHA_VERSION: ${{secrets.AWS_S3_ACCESS_KEY && 0 || 1}}
SCCACHE_BUCKET: arcticdb-ci-sccache-bucket
SCCACHE_ENDPOINT: http://s3.eu-west-1.amazonaws.com
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know what region our CI runners run in? Allegedly they're all in the US, so that might be a better location for the cache.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will test this next week

SCCACHE_REGION: eu-west-1
SCCACHE_S3_USE_SSL: false
AWS_ACCESS_KEY_ID: ${{secrets.AWS_S3_ACCESS_KEY}}
AWS_SECRET_ACCESS_KEY: ${{secrets.AWS_S3_SECRET_KEY}}
VCPKG_NUGET_USER: ${{secrets.VCPKG_NUGET_USER || github.repository_owner}}
VCPKG_NUGET_TOKEN: ${{secrets.VCPKG_NUGET_TOKEN || secrets.GITHUB_TOKEN}}
VCPKG_MAN_NUGET_USER: ${{secrets.VCPKG_MAN_NUGET_USER}} # For forks to download pre-compiled dependencies from the Man repo
Expand All @@ -63,6 +71,7 @@ jobs:
VCPKG_BINARY_SOURCES VCPKG_NUGET_USER VCPKG_NUGET_TOKEN VCPKG_MAN_NUGET_USER VCPKG_MAN_NUGET_TOKEN
CMAKE_C_COMPILER_LAUNCHER CMAKE_CXX_COMPILER_LAUNCHER CMAKE_BUILD_PARALLEL_LEVEL ARCTIC_CMAKE_PRESET
ARCTICDB_BUILD_DIR TEST_OUTPUT_DIR ARCTICDB_VCPKG_INSTALLED_DIR ARCTICDB_VCPKG_PACKAGES_DIR
SCCACHE_BUCKET SCCACHE_ENDPOINT AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY SCCACHE_REGION SCCACHE_S3_USE_SSL
ARCTICDB_DEBUG_FIND_PYTHON: ${{vars.ARCTICDB_DEBUG_FIND_PYTHON}}
python_impl_name: ${{inputs.python3 > 0 && format('cp3{0}', inputs.python3) || 'default'}}
CIBW_BUILD: ${{format('cp3{0}-{1}', inputs.python3, matrix.cibw_build_suffix)}}
Expand Down Expand Up @@ -101,16 +110,12 @@ jobs:
CMAKE_BUILD_PARALLEL_LEVEL: ${{vars.CMAKE_BUILD_PARALLEL_LEVEL}}

# ========================= Leader steps =========================
- name: Remove GitHub default packages (baseimage) # To save space
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is no longer needed as the cache is reducing the compilation load, i.e. no as much disk space is needed.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

May still need it if we ever tear down (or invalidate) the whole cache?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem like it, the wheels build was able to run even when an empty cache (when I changed to S3)
The step below is still needed for C++ tests jobs.

if: inputs.job_type == 'build-python-wheels' && matrix.os == 'linux'
uses: jlumbroso/free-disk-space@main

- name: Remove GitHub default packages (manylinux) # To save space
if: inputs.job_type == 'cpp-tests' && matrix.os == 'linux'
run: |
du -m /mnt/usr/local/lib/ | sort -n | tail -n 50
nohup rm -rf /mnt/usr/local/lib/android &

- name: Find and remove ccache # See PR: #945
if: matrix.os == 'windows'
run: rm $(which ccache) || true
Expand Down
1 change: 0 additions & 1 deletion cpp/CMakePresets.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@
"generator": "Ninja",
"environment": { "cmakepreset_expected_host_system": "Windows" },
"cacheVariables": {
"ARCTICDB_USE_PCH": "ON",
"VCPKG_OVERLAY_TRIPLETS": "custom-triplets",
"VCPKG_TARGET_TRIPLET": "x64-windows-static-msvc"
}
Expand Down
Loading