Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DRAFT: llvm artefact management #704

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

DRAFT: llvm artefact management #704

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .github/actions/clean_cache/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,8 @@ runs:
# key is first argument
for CACHE_PREFIX in ${{ inputs.cache_prefixes }}
do
grep -E -o "${CACHE_PREFIX}[^[:space:]]+" CACHE_LIST | sed '1d' | awk '{ print $1 }'
# we don't want the error code if we don't find it the cache.
(grep -E -o "${CACHE_PREFIX}[^[:space:]]+" CACHE_LIST || true) | sed '1d' | awk '{ print $1 }'
done > CCACHE_LIST

cat CCACHE_LIST
Expand Down
218 changes: 218 additions & 0 deletions .github/workflows/create_llvm_artefacts.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,218 @@
# Simple workflow for running non-documentation PR testing
name: cache llvm artefacts
on:
workflow_call:
inputs:
llvm_branch:
required: false
type: string
default: 'release/19.x'
llvm_branch_id:
required: false
type: string
default: '19'
# outputs:
# workflow_id:
# value: ${{ github.run_id}}

permissions: {}

jobs:
create_llvm_artifacts:
strategy:
matrix:
# os: [ubuntu-22.04, ubuntu-24.04, windows-2019]
# build_type: [RelAssert]
# arch : [x86, x86_64, aarch64, riscv64]
# exclude:
# Reduce the versions of llvm for aarch64, riscv64 & x86 architectures and windows, as
# well as any windows os variants
# - os: windows-2019
# arch: aarch64
# - os: windows-2019
# arch: riscv64
# - os: windows-2019
# arch: x86

# - os: ubuntu-22.04
# arch: riscv64
# - os: ubuntu-24.04
# arch: aarch64
# - os: ubuntu-24.04
# arch: x86
# include:
# # We want to set flags related to particular matrix dimensions. To do this
# # we need to create default values first, and then against particular matrix
# # dimensions.
# # Note that we need to use RelAssert as the cache key matching can match Release against ReleaseAssert
# - os_flags:
# - arch_flags:
# - build_type_flags:
# - build_type_flags: -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
# build_type: RelAssert
# - arch_flags: -DCMAKE_TOOLCHAIN_FILE="$GITHUB_WORKSPACE/ock/platform/arm-linux/aarch64-toolchain.cmake"
# -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu
# arch: aarch64
# - arch_flags: -DCMAKE_TOOLCHAIN_FILE="$GITHUB_WORKSPACE/ock/platform/riscv64-linux/riscv64-gcc-toolchain.cmake"
# -DLLVM_HOST_TRIPLE=riscv64-unknown-linux-gnu
# arch: riscv64
# - arch_flags: -DLLVM_BUILD_32_BITS=ON -DLIBXML2_LIBRARIES=IGNORE -DLLVM_ENABLE_TERMINFO=OFF
# -DLLVM_HOST_TRIPLE=i686-unknown-linux-gnu
# arch: x86
# - os: ubuntu-22.04
# os_flags: -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
# - os: ubuntu-24.04
# os_flags: -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
os: [ubuntu-22.04]
build_type: [RelAssert]
arch : [x86_64]
# exclude:
# Reduce the versions of llvm for aarch64, riscv64 & x86 architectures and windows, as
# well as any windows os variants
# - os: windows-2019
# arch: aarch64
# - os: windows-2019
# arch: riscv64
# - os: windows-2019
# arch: x86

# - os: ubuntu-22.04
# arch: riscv64
# - os: ubuntu-24.04
# arch: aarch64
# - os: ubuntu-24.04
# arch: x86
include:
# We want to set flags related to particular matrix dimensions. To do this
# we need to create default values first, and then against particular matrix
# dimensions.
# Note that we need to use RelAssert as the cache key matching can match Release against ReleaseAssert
- os_flags:
- arch_flags:
- build_type_flags:
- build_type_flags: -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
build_type: RelAssert
# - arch_flags: -DCMAKE_TOOLCHAIN_FILE="$GITHUB_WORKSPACE/ock/platform/arm-linux/aarch64-toolchain.cmake"
# -DLLVM_HOST_TRIPLE=aarch64-unknown-linux-gnu
# arch: aarch64
# - arch_flags: -DCMAKE_TOOLCHAIN_FILE="$GITHUB_WORKSPACE/ock/platform/riscv64-linux/riscv64-gcc-toolchain.cmake"
# -DLLVM_HOST_TRIPLE=riscv64-unknown-linux-gnu
# arch: riscv64
# - arch_flags: -DLLVM_BUILD_32_BITS=ON -DLIBXML2_LIBRARIES=IGNORE -DLLVM_ENABLE_TERMINFO=OFF
# -DLLVM_HOST_TRIPLE=i686-unknown-linux-gnu
# arch: x86
- os: ubuntu-22.04
os_flags: -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
# - os: ubuntu-24.04
# os_flags: -DLLVM_BUILD_LLVM_DYLIB=ON -DLLVM_LINK_LLVM_DYLIB=ON
runs-on: ${{ matrix.os }}
steps:

- name: Setup Windows
if: startsWith(runner.os, 'Windows') && steps.cache.outputs.cache-hit != 'true'
uses: llvm/actions/setup-windows@main
with:
arch: amd64

- name: Checkout repo llvm
if: steps.cache.outputs.cache-hit != 'true'
uses: actions/checkout@v4
with:
repository: llvm/llvm-project
ref: ${{inputs.llvm_branch}}

- name: Install Ninja
uses: llvm/actions/install-ninja@main

- name: Setup ccache
if: matrix.arch == 'x86_64'
uses: hendrikmuhs/ccache-action@53911442209d5c18de8a31615e0923161e435875 # v1.2.16
with:
max-size: 200M
key: ccache-llvm-build-${{ inputs.os }}
variant: ccache
# TODO: How do we handle that these are parallel?
save: true

- name: Checkout repo ock platform
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.arch != 'x86_64' }}
uses: actions/checkout@v4
with:
sparse-checkout: |
platform
path: ock

- name: Install Ninja
if: steps.cache.outputs.cache-hit != 'true'
uses: llvm/actions/install-ninja@a1ea791b03c8e61f53a0e66f2f73db283aa0f01e # main branch

- name: install aarch64 build tools
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.arch == 'aarch64' }}
run:
sudo apt-get install --yes g++-aarch64-linux-gnu

- name: install riscv64 build tools
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.arch == 'riscv64' }}
run:
sudo apt-get install --yes g++-riscv64-linux-gnu

- name: install x86 build tools
if: ${{ steps.cache.outputs.cache-hit != 'true' && matrix.arch == 'x86' }}
run: |
sudo dpkg --add-architecture i386
sudo apt-get update
sudo apt-get install --yes gcc-multilib g++-multilib libc6-dev:i386 lib32tinfo-dev

- name: Flags checker
if: steps.cache.outputs.cache-hit != 'true'
run:
echo Building on "${{ matrix.os }}" with os_flags "${{ matrix.os_flags}}" arch "${{ matrix.arch }}" extra flags "${{ matrix.build_type_flags}}" and build_type "${{matrix.build_type}}"

- name: Run cmake
if: steps.cache.outputs.cache-hit != 'true'
run:
cmake llvm
-DLLVM_ENABLE_DIA_SDK=OFF
-DCMAKE_INSTALL_PREFIX=llvm_install
-DLLVM_ENABLE_ZLIB=FALSE
-DLLVM_ENABLE_ZSTD=FALSE
-DLLVM_ENABLE_Z3_SOLVER=FALSE
-DLLVM_ENABLE_PROJECTS="clang;lld"
-DLLVM_TARGETS_TO_BUILD="X86;ARM;AArch64;RISCV"
-DLLVM_CCACHE_BUILD=ON
-Bbuild
-GNinja
${{ matrix.build_type_flags }}
${{ matrix.os_flags}}
${{ matrix.arch_flags}}

- name: Run build on llvm
run:
cmake --build build --target install

- name: Copy lit tools
run: |
cp build/bin/FileCheck* llvm_install/bin
cp build/bin/not* llvm_install/bin

- name: upload artefact
uses: actions/upload-artifact@v4
with:
name: llvm-${{ matrix.os }}-${{ matrix.arch }}-${{ inputs.llvm_branch_id }}-${{ matrix.build_type }}
path: llvm_install
retention-days: 7


delete_llvm_build_cache:
needs: [ create_llvm_artifacts ]
runs-on: ubuntu-latest
steps:
- name: Checkout repo
uses: actions/checkout@v4
with:
sparse-checkout: .github
- name: Cache clean
uses: ./.github/actions/clean_cache
with:
token: ${{ secrets.GITHUB_TOKEN }}
cache_prefixes: "ccache-llvm-build-ubuntu-22.04 ccache-llvm-build-ubuntu-24.04 ccache-llvm-build-windows-2019"
44 changes: 44 additions & 0 deletions .github/workflows/create_llvm_artefacts_caller.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# Temporary for testing - move to planned_testing_caller.yml
name: Run ock tests for PR style testing
on:
pull_request:
paths:
- '.github/workflows/create_llvm_artefacts.yml'
- '.github/workflows/create_llvm_artefacts_caller.yml'
schedule:
# Run Mon-Fri at 7pm
- cron: '00 19 * * 1-5'

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true

jobs:
build_llvm_artefacts:
name: Call PR testing on schedule
if: ${{ (github.event_name == 'schedule' && github.repository == 'uxlfoundation/oneapi-construction-kit') || github.event_name == 'pull_request' }}
permissions:
actions: write
uses: ./.github/workflows/create_llvm_artefacts.yml
secrets: inherit
with:
llvm_branch: 'release/19.x'
llvm_branch_id: '19'

use_llvm_artefacts:
needs: [build_llvm_artefacts]
runs-on: ubuntu-22.04
steps:
- name: Checkout repo
uses: actions/checkout@v4
- name: set up gh
uses: ./.github/actions/setup_gh
with:
os: ${{ contains( matrix.target, 'windows') && 'windows' || 'ubuntu' }}
token: ${{ secrets.GITHUB_TOKEN }}
- name: Try downloading llvm install
run: |
echo workflow id is ${{needs.build_llvm_artefacts.outputs.workflow_id}}
# TODO: We need to add tarring there and here
gh run download ${{github.run_id}} -n llvm-ubuntu-22.04-x86_64-19-RelAssert -D llvm_install
ls -l llvm_install/bin
Loading