Fix building on Alpine Linux and musl

[yorickpeterse]

Using musl would allow Inko executables to not depend on glibc, making them more portable. Alpine Linux in turn is commonly used in CI environments.

Support for Alpine Linux is currently blocked by the following issues:

• https://gitlab.com/taricorp/llvm-sys.rs/-/issues/44
• page_size() returns zero under Alpine Linux bytecodealliance/rustix#933

Outside of that, it seems to work, but it requires some effort: you have to build the compiler on a non-musl target, then use that to compile Inko to object files. You then have to build the runtime (using cargo build -p rt) under Alpine, and manually link things together (again using Alpine) like so:

cc build/objects/*.o target/x86_64-unknown-linux-musl/debug/libinko.a -static -fuse-ld=lld -o /tmp/test

Without the -static flag there's still some dynamic linking going on, probably due to Alpine dynamically linking musl by default:

ldd /tmp/test
/lib/ld-musl-x86_64.so.1 (0x7f07848ff000)
libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x7f07843f9000)
libc.musl-x86_64.so.1 => /lib/ld-musl-x86_64.so.1 (0x7f07848ff000)

You then need the following patch to work around the rustix bug:

diff --git a/rt/src/page.rs b/rt/src/page.rs
index 2cb97a22..624d765d 100644
--- a/rt/src/page.rs
+++ b/rt/src/page.rs
@@ -1,11 +1,16 @@
+use libc::{sysconf, _SC_PAGESIZE};
 use std::sync::atomic::{AtomicUsize, Ordering};
 
 static PAGE_SIZE: AtomicUsize = AtomicUsize::new(0);
 
+fn page_size_raw() -> usize {
+    unsafe { sysconf(_SC_PAGESIZE) as usize }
+}
+
 pub(crate) fn page_size() -> usize {
     match PAGE_SIZE.load(Ordering::Relaxed) {
         0 => {
-            let size = rustix::param::page_size();
+            let size = page_size_raw();
 
             PAGE_SIZE.store(size, Ordering::Relaxed);
             size

Once that's all done, the resulting executable works fine.

Once the mentioned bugs are solved, we'll need to extend the compiler with a proper musl target, as there may be some differences in LLVM between XXX-unknown-linux-musl and XXX-unknown-linux-gnu. We'll also need to adjust the linker flags for Alpine, as many of the current flags aren't needed and cause linker errors.

Related issues

• Provide a way of linking libgcc statically rust-lang/rust#29527
• staticlib libgcc_s dependency requirement since 1.21.0 (nightly-2017-08-24) rust-lang/rust#55120
• -crt-static on musl links with libgcc_s rust-lang/rust#82521
• Android NDK r25b changes will break developers using r22b or older rust-lang/rust#103673
• Rustup should warn if libgcc is not installed on Alpine rust-lang/rustup#2213
• rust: cross build broken on armv*hf-musl void-linux/void-packages#3605
• Link against LLVM libunwind (statically) instead of libgcc_s (dynamically) on Linux targets rust-lang/rust#119504

[yorickpeterse]

To add to that: building Rust static libraries that target musl on a GNU host is straight up broken, and has been since 2018 or so (see this issue, there are a few more but I can't remember their titles/URLs). Basically the issue is that when you do cargo build -p rt --target x86_64-unknown-linux-musl on a GNU host, the resulting library still tries to link against some libgcc library/functions that aren't available under musl, resulting in linker errors. Fiddling with Rust's crt-static flags doesn't help either, as IIRC it's the result of some #[cfg(..)] block making the wrong decision for this particular setup.

We can alleviate that pain a bit with #524 by compiling the runtime ahead of time, but it's something we'd have to document clearly in case somebody wants to build the runtime from source, on a GNU host, but for a musl target.

[yorickpeterse]

We can now sort of build things for musl as follows:

cargo build -p rt --target x86_64-unknown-linux-musl # Build just the runtime for musl
cargo build # Build the compiler for the host
cp target/x86_64-unknown-linux-musl/release/libinko.a target/debug/libinko-amd64-linux-musl.a
./target/debug/inko build --target=amd64-linux-musl -o /tmp/test ~/Downloads/test.inko 
ldd /tmp/test
        linux-vdso.so.1 (0x00007ffe283cb000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007ff4ed68a000)
        /lib/ld-musl-x86_64.so.1 => /usr/lib64/ld-linux-x86-64.so.2 (0x00007ff4eda54000)

However, if we run the resulting executable we get this:

Error relocating /lib/ld-linux-x86-64.so.2: unsupported relocation type 37
Error relocating /tmp/test: __pthread_key_create: symbol not found
Error relocating /tmp/test: __register_atfork: symbol not found
Error relocating /tmp/test: __cxa_thread_atexit_impl: symbol not found
Error relocating /tmp/test: _dl_find_object: symbol not found

To "fix" that we have to build like so:

./target/debug/inko build --target=amd64-linux-musl -o /tmp/test ~/Downloads/test.inko --linker-arg='-static'

The resulting executable is then statically linked, and runs just fine.

However, I strongly suspect that what this actually does is statically link to GNU libc and not the musl libc, because this works even if you don't have any musl libraries installed.

Using musl-clang as the linker we get errors such as this:

./target/debug/inko build --target=amd64-linux-musl -o /tmp/test ~/Downloads/test.inko --linker=musl-clang
error: the linker exited with status code 1:

clang-16: warning: argument unused during compilation: '-nostdinc' [-Wunused-command-line-argument]
/usr/sbin/ld: /usr/sbin/../lib64/gcc/x86_64-pc-linux-gnu/13.2.1/libgcc_eh.a(unwind-dw2-fde-dip.o): in function `_Unwind_Find_FDE':
(.text+0x250d): undefined reference to `_dl_find_object'
clang-16: error: linker command failed with exit code 1 (use -v to see invocation)

The musl source code in turn has no reference of _dl_find_object, resulting in this linker error.

Unfortunately, what little info I can find points back to the linked Rust issues, so I'm not sure what the solution here is.

[yorickpeterse]

https://github.com/nbdd0121/unwinding seems interesting in that it claims to be a pure-Rust replacement for libgcc_eh & friends. Unfortunately, it requires a nightly version of Rust, and it's not clear when it could be used on a stable version.

With that said, it does seem to work if we use musl-clang as the linker while using this library. For example, this program:

fn foo {
  bar
}

fn bar {
  baz
}

fn baz {
  quix
}

fn quix {
  panic('testing')
}

class async Main {
  fn async main {
    foo
  }
}

Produces:

Stack trace (the most recent call comes last):
  /var/home/yorickpeterse/Downloads/test.inko:21 in main.Main.main
  /var/home/yorickpeterse/Downloads/test.inko:4 in main.foo
  /var/home/yorickpeterse/Downloads/test.inko:8 in main.bar
  /var/home/yorickpeterse/Downloads/test.inko:12 in main.baz
  /var/home/yorickpeterse/Downloads/test.inko:16 in main.quix
  /var/home/yorickpeterse/Projects/inko/inko/std/src/std/process.inko:15 in std.process.panic
Process 'Main' (0x56467e3daf40) panicked: testing

It also works if we set the sysroot and use regular clang:

./target/debug/inko build --target=amd64-linux-musl -o /tmp/test ~/Downloads/test.inko --linker=clang --linker-arg='--sysroot=/usr/lib/musl'

This however will fail if you try to link additional libraries as those won't be located in /usr/lib/musl, so I'm not sure what the best option is there. With that said, this also seems to happen when using musl-clang, so it's probably a more generic issue with cross-compiling.

[yorickpeterse]

Per rust-lang/rust@ee870d6, it seems progress is being made towards including support for the unwinding crate in Rust itself. This means that at some point in the future, we may not need gcc_s/etc any more, but it's not clear when that would be.

[yorickpeterse]

I've tried various approaches to building the static library such that gcc_eh isn't included, but it seems that no matter what combination of distribution/RUSTFLAGS/etc we use, Rust insists on including the library, and no matter what I try I can't get this to work with musl-clang. The result is that it's still not yet possible to build using musl :<

[yorickpeterse]

For #524 there is a sort of workaround: we can build the runtime using a nightly version of Rust using the "unwinding" crate, using this patch:

diff --git a/rt/Cargo.toml b/rt/Cargo.toml
index b730832c..9f442973 100644
--- a/rt/Cargo.toml
+++ b/rt/Cargo.toml
@@ -20,6 +20,7 @@ polling = "^2.8"
 unicode-segmentation = "^1.8"
 backtrace = "^0.3"
 rustix = { version = "^0.38.24", features = ["fs", "mm", "param", "process", "net", "std", "time"], default-features = false }
+unwinding = "*"
 
 [dependencies.socket2]
 version = "^0.5"
diff --git a/rt/src/lib.rs b/rt/src/lib.rs
index cdc264fb..e5fb0b14 100644
--- a/rt/src/lib.rs
+++ b/rt/src/lib.rs
@@ -3,6 +3,8 @@
 #![cfg_attr(feature = "cargo-clippy", allow(clippy::missing_safety_doc))]
 #![cfg_attr(feature = "cargo-clippy", allow(clippy::too_many_arguments))]
 
+extern crate unwinding;
+
 pub mod macros;
 
 pub mod arc_without_weak;

We'd then build the runtime ahead of time like so:

cargo +nightly build -p rt --release --target=x86_64-unknown-linux-musl

You can then compile the compiler using stable Rust and use this nightly-built runtime for musl, removing the need for gcc_eh & friends.

However, until https://gitlab.com/taricorp/llvm-sys.rs/-/issues/44 is taken care of I'm not sure what happens if you try to build on Alpine for Alpine. This approach also means that we technically still have to depend on libgcc, at least until the unwinding crate is available on stable versions of Rust.

[yorickpeterse]

It seems the use of the "unwinding" crate produces somewhat inconsistent results: when linking using musl-clang, a panic produces a stacktrace. When linking with musl-gcc, no stacktrace is produced. Playing around with the different feature flags doesn't seem to fix this.

Based on this, I'm going to abandon the idea of using the "unwinding" crate, as it feels like something that's just too unstable/inconsistent for our usecase at this point.

[yorickpeterse]

To add to the above:

The crux of the issue is that unwinding requires certain _Unwind_XXX functions to be defined. When linking with musl, even when the "unwinding" crate is used, musl-clang/musl-gcc injects -lgcc_eh albeit with a --as-needed flag. Because "unwinding" defines these functions, the linking of gcc_eh is ignored as it's deemed unnecessary.

Linking against libunwind could solve that as it defines these functions, but at least on my platform (Arch Linux) I seemingly can't get it to either dynamically link (it just seemingly ignores it) or statically link (as Arch doesn't ship with static libraries).

Thus, I wonder if perhaps the fact that I'm using Arch Linux plays a role here, so I'll see what happens if I link on e.g. Fedora.

[yorickpeterse]

Fedora exhibits the same issues, so this isn't a distribution related issue.

[yorickpeterse]

I sort of made some more (interesting) progress: Rust's musl target bundles a static libunwind library in ~/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/libunwind.a. If I inject that as an argument after the runtime, I can at least compile/link generated code that targets musl:

diff --git a/compiler/src/linker.rs b/compiler/src/linker.rs
index a1ceec8d..f13c9d71 100644
--- a/compiler/src/linker.rs
+++ b/compiler/src/linker.rs
@@ -233,6 +233,7 @@ pub(crate) fn link(
     })?;
 
     cmd.arg(&rt_path);
+    cmd.arg("/var/home/yorickpeterse/homes/arch/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/libunwind.a");
 
     // Include any extra platform specific libraries, such as libm on the
     // various Unix platforms. These must come _after_ any object files and

Oddly enough, if I link using musl-clang I get a working stack trace for a panic. If instead musl-gcc is used, the trace is missing. ldd also reports "error while loading shared libraries: /usr/lib/libc.so: invalid ELF header", suggesting we may need to pass some additional flags.

[yorickpeterse]

https://github.com/rust-lang/rust/blob/62d7ed4a6775c4490e493093ca98ef7c215b835b/compiler/rustc_codegen_ssa/src/back/linker.rs#L775 suggests we need to pass --eh-frame-hdr when using musl-gcc. Removing this option from the musl-clang output reproduces the lack of a stack trace.

[yorickpeterse]

For musl, what we could do is this: the runtimes we build aren't just gzipped .a files, but .tar.gz archives. We then extract those into ~/.local/share/inko/VERSION/runtimes/TARGET/. For musl, we include the libunwind.a as provided by the Rust musl toolchain (so we ensure the version is compatible with Rust) for musl targets. We then include said libunwind.a on non-musl hosts when targeting musl. On musl targets we probably don't need to do so, as Alpine seems to actually prefer the system-wide libgcc_whatever (IIRC they patch Rust to achieve that).

[yorickpeterse]

I got linking with musl to work, so we can at least build executables that target musl now. Building the compiler on Apline is still broken though, as https://gitlab.com/taricorp/llvm-sys.rs/-/issues/44 hasn't been solved yet.

[yorickpeterse]

Per https://gitlab.com/taricorp/llvm-sys.rs/-/issues/44#note_2095470233, we should soon be able to build on Alpine once more.