Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build musl wheels #155

Merged
merged 35 commits into from
Nov 3, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
d54c980
build musl wheels
Congyuwang Nov 2, 2024
2448699
attempt to fix macos build
Congyuwang Nov 2, 2024
02e4d5c
try to build macos
Congyuwang Nov 2, 2024
bca692e
attempt to build musllinux
Congyuwang Nov 2, 2024
421d51b
amend
Congyuwang Nov 2, 2024
7170a0d
try building with static bindgen
Congyuwang Nov 2, 2024
e6ae9f8
only musl in musl CI and PR
Congyuwang Nov 2, 2024
46c2a65
Merge branch 'main' into musl-wheels
Congyuwang Nov 2, 2024
e376f47
just to push again
Congyuwang Nov 2, 2024
fb18429
Revert "just to push again"
Congyuwang Nov 2, 2024
1c217c9
install zlib and ncurses
Congyuwang Nov 2, 2024
b816977
try libc++
Congyuwang Nov 3, 2024
1d671e1
Revert "try libc++"
Congyuwang Nov 3, 2024
afceb4a
include gcc header search path
Congyuwang Nov 3, 2024
9f46976
try again
Congyuwang Nov 3, 2024
3f098b7
try again
Congyuwang Nov 3, 2024
35436de
detailed isystem
Congyuwang Nov 3, 2024
82a9f84
remove aarch-gcc-headers
Congyuwang Nov 3, 2024
3445f4e
fix missing bits headers
Congyuwang Nov 3, 2024
ff16286
add LDFLAGS
Congyuwang Nov 3, 2024
3edd503
try to fix missing crt library
Congyuwang Nov 3, 2024
b7054ce
fix missing crt objects
Congyuwang Nov 3, 2024
a824cd5
add gcc path
Congyuwang Nov 3, 2024
2af199f
try using soft-link for libgcc
Congyuwang Nov 3, 2024
a1531e4
run test in container
Congyuwang Nov 3, 2024
1ec6b95
add test folder
Congyuwang Nov 3, 2024
b2d4350
fix tests
Congyuwang Nov 3, 2024
cd03a3e
fix pip install
Congyuwang Nov 3, 2024
1bf15aa
try to fix tests
Congyuwang Nov 3, 2024
7ed2473
fix the default-features of librocksdb-sys
Congyuwang Nov 3, 2024
3d91839
add all wheels back
Congyuwang Nov 3, 2024
ba96059
just to push
Congyuwang Nov 3, 2024
252cef4
Revert "just to push"
Congyuwang Nov 3, 2024
1c394b6
remove static bindgen in Cargo.toml
Congyuwang Nov 3, 2024
879e141
update rust-rocksdb branch
Congyuwang Nov 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 99 additions & 1 deletion .github/workflows/CI.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CI
name: CI musl

on:
release:
Expand Down Expand Up @@ -278,6 +278,102 @@ jobs:
name: wheels-manylinux2014-${{ matrix.target }}
path: dist

musllinux_1_2_x86_64:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
target: [x86_64]
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Build Wheels
uses: messense/maturin-action@v1
with:
rust-toolchain: 1.82.0
rustup-components: rustfmt
target: ${{ matrix.target }}
manylinux: musllinux_1_2
args: --release --no-default-features --features bindgen-static --out dist --interpreter '3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.9 pypy3.10'
container: congyuwang/musllinux_1_2_x86_64:llvm-19.1.3
before-script-linux: |
apk update
apk add ncurses-static
apk add zlib-static
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/crtbeginS.o /usr/lib/crtbeginS.o
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/crtendS.o /usr/lib/crtendS.o
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/libgcc.a /usr/lib/libgcc.a
env:
LIBCLANG_PATH: /usr/local/lib
CC: /usr/local/bin/clang
CXX: /usr/local/bin/clang++
AR: /usr/local/bin/llvm-ar
CFLAGS: "-flto=thin -O3 -isystem/usr/include"
CXXFLAGS: "-flto=thin -O3 -isystem/usr/include/c++/13.2.1/ -isystem/usr/include -isystem/usr/include/c++/13.2.1/x86_64-alpine-linux-musl"
RUSTFLAGS: "-Clinker-plugin-lto -Clinker=/usr/local/bin/clang -Clink-arg=-fuse-ld=/usr/local/bin/ld.lld"
LLVM_CONFIG_PATH: /root/llvm_config.sh
LIBCLANG_STATIC_PATH: /usr/local/lib
- name: Run Tests
run: |
docker run --rm -v ${{ github.workspace }}/dist:/dist:ro -v ${{ github.workspace }}/test:/test:ro congyuwang/musllinux_1_2_x86_64:llvm-19.1.3 bash -c "
python3.12 -m pip install rocksdict --no-index --find-links /dist --force-reinstall || true
python3.12 -m pip install speedict --no-index --find-links /dist --force-reinstall || true
python3.12 -m unittest discover -v /test"
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: wheels-musllinux_1_2-x86_64
path: dist

musllinux_1_2_arm64:
runs-on: linux-arm64
strategy:
fail-fast: false
matrix:
target: [aarch64]
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Build Wheels
uses: messense/maturin-action@v1
with:
rust-toolchain: 1.82.0
rustup-components: rustfmt
target: ${{ matrix.target }}
manylinux: musllinux_1_2
args: --release --no-default-features --features bindgen-static --out dist --interpreter '3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.9 pypy3.10'
container: congyuwang/musllinux_1_2_aarch64:llvm-19.1.3
before-script-linux: |
apk update
apk add ncurses-static
apk add zlib-static
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/crtbeginS.o /usr/lib/crtbeginS.o
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/crtendS.o /usr/lib/crtendS.o
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/libgcc.a /usr/lib/libgcc.a
env:
LIBCLANG_PATH: /usr/local/lib
CC: /usr/local/bin/clang
CXX: /usr/local/bin/clang++
AR: /usr/local/bin/llvm-ar
CFLAGS: "-flto=thin -O3 -isystem/usr/include"
CXXFLAGS: "-flto=thin -O3 -isystem/usr/include/c++/13.2.1/ -isystem/usr/include -isystem/usr/include/c++/13.2.1/aarch64-alpine-linux-musl"
RUSTFLAGS: "-Clinker-plugin-lto -Clinker=/usr/local/bin/clang -Clink-arg=-fuse-ld=/usr/local/bin/ld.lld"
LLVM_CONFIG_PATH: /root/llvm_config.sh
LIBCLANG_STATIC_PATH: /usr/local/lib
- name: Run Tests
run: |
docker run --rm -v ${{ github.workspace }}/dist:/dist:ro -v ${{ github.workspace }}/test:/test:ro congyuwang/musllinux_1_2_aarch64:llvm-19.1.3 bash -c "
python3.12 -m pip install rocksdict --no-index --find-links /dist --force-reinstall || true
python3.12 -m pip install speedict --no-index --find-links /dist --force-reinstall || true
python3.12 -m unittest discover -v /test"
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: wheels-musllinux_1_2-arm64
path: dist

release:
name: Release
runs-on: ubuntu-latest
Expand All @@ -289,6 +385,8 @@ jobs:
manylinux2014_x86_64,
manylinux_2_28_arm64,
manylinux2014_arm64,
musllinux_1_2_x86_64,
musllinux_1_2_arm64,
]
steps:
- uses: actions/download-artifact@v4
Expand Down
98 changes: 97 additions & 1 deletion .github/workflows/PR.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: PR check
name: PR check musl

on:
pull_request:
Expand Down Expand Up @@ -277,3 +277,99 @@ jobs:
with:
name: wheels-manylinux2014-${{ matrix.target }}
path: dist

musllinux_1_2_x86_64:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
target: [x86_64]
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Build Wheels
uses: messense/maturin-action@v1
with:
rust-toolchain: 1.82.0
rustup-components: rustfmt
target: ${{ matrix.target }}
manylinux: musllinux_1_2
args: --release --no-default-features --features bindgen-static --out dist --interpreter '3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.9 pypy3.10'
container: congyuwang/musllinux_1_2_x86_64:llvm-19.1.3
before-script-linux: |
apk update
apk add ncurses-static
apk add zlib-static
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/crtbeginS.o /usr/lib/crtbeginS.o
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/crtendS.o /usr/lib/crtendS.o
ln -s /usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/libgcc.a /usr/lib/libgcc.a
env:
LIBCLANG_PATH: /usr/local/lib
CC: /usr/local/bin/clang
CXX: /usr/local/bin/clang++
AR: /usr/local/bin/llvm-ar
CFLAGS: "-flto=thin -O3 -isystem/usr/include"
CXXFLAGS: "-flto=thin -O3 -isystem/usr/include/c++/13.2.1/ -isystem/usr/include -isystem/usr/include/c++/13.2.1/x86_64-alpine-linux-musl"
RUSTFLAGS: "-Clinker-plugin-lto -Clinker=/usr/local/bin/clang -Clink-arg=-fuse-ld=/usr/local/bin/ld.lld"
LLVM_CONFIG_PATH: /root/llvm_config.sh
LIBCLANG_STATIC_PATH: /usr/local/lib
- name: Run Tests
run: |
docker run --rm -v ${{ github.workspace }}/dist:/dist:ro -v ${{ github.workspace }}/test:/test:ro congyuwang/musllinux_1_2_x86_64:llvm-19.1.3 bash -c "
python3.12 -m pip install rocksdict --no-index --find-links /dist --force-reinstall || true
python3.12 -m pip install speedict --no-index --find-links /dist --force-reinstall || true
python3.12 -m unittest discover -v /test"
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: wheels-musllinux_1_2-x86_64
path: dist

musllinux_1_2_arm64:
runs-on: linux-arm64
strategy:
fail-fast: false
matrix:
target: [aarch64]
steps:
- uses: actions/checkout@v2
with:
submodules: recursive
- name: Build Wheels
uses: messense/maturin-action@v1
with:
rust-toolchain: 1.82.0
rustup-components: rustfmt
target: ${{ matrix.target }}
manylinux: musllinux_1_2
args: --release --no-default-features --features bindgen-static --out dist --interpreter '3.7 3.8 3.9 3.10 3.11 3.12 3.13 pypy3.9 pypy3.10'
container: congyuwang/musllinux_1_2_aarch64:llvm-19.1.3
before-script-linux: |
apk update
apk add ncurses-static
apk add zlib-static
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/crtbeginS.o /usr/lib/crtbeginS.o
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/crtendS.o /usr/lib/crtendS.o
ln -s /usr/lib/gcc/aarch64-alpine-linux-musl/13.2.1/libgcc.a /usr/lib/libgcc.a
env:
LIBCLANG_PATH: /usr/local/lib
CC: /usr/local/bin/clang
CXX: /usr/local/bin/clang++
AR: /usr/local/bin/llvm-ar
CFLAGS: "-flto=thin -O3 -isystem/usr/include"
CXXFLAGS: "-flto=thin -O3 -isystem/usr/include/c++/13.2.1/ -isystem/usr/include -isystem/usr/include/c++/13.2.1/aarch64-alpine-linux-musl"
RUSTFLAGS: "-Clinker-plugin-lto -Clinker=/usr/local/bin/clang -Clink-arg=-fuse-ld=/usr/local/bin/ld.lld"
LLVM_CONFIG_PATH: /root/llvm_config.sh
LIBCLANG_STATIC_PATH: /usr/local/lib
- name: Run Tests
run: |
docker run --rm -v ${{ github.workspace }}/dist:/dist:ro -v ${{ github.workspace }}/test:/test:ro congyuwang/musllinux_1_2_aarch64:llvm-19.1.3 bash -c "
python3.12 -m pip install rocksdict --no-index --find-links /dist --force-reinstall || true
python3.12 -m pip install speedict --no-index --find-links /dist --force-reinstall || true
python3.12 -m unittest discover -v /test"
- name: Upload artifacts
uses: actions/upload-artifact@v4
with:
name: wheels-musllinux_1_2-arm64
path: dist
15 changes: 13 additions & 2 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -16,13 +16,24 @@ path = "bin/create-cf-db/main.rs"
test = false

[dependencies]
rocksdb = { path = "rust-rocksdb" }
librocksdb-sys = { path = "rust-rocksdb/librocksdb-sys" }
rocksdb = { path = "rust-rocksdb", default-features = false, features = [
"snappy",
"lz4",
"zstd",
"zlib",
"bzip2",
] }
librocksdb-sys = { path = "rust-rocksdb/librocksdb-sys", default-features = false }
serde = { version = "1", features = ["derive"] }
serde_json = "1"
num-bigint = "0.4"
libc = "0.2"

[features]
default = ["bindgen-runtime"]
bindgen-runtime = ["rocksdb/bindgen-runtime"]
bindgen-static = ["rocksdb/bindgen-static"]

[dependencies.pyo3]
version = "0.22"
features = ["py-clone", "extension-module", "num-bigint"]
Expand Down
2 changes: 1 addition & 1 deletion rust-rocksdb
Loading