Link-method / format for Rust plugins that statically link libc

[rrnewton]

It is a long term goal to provide libc / standard library functionality for writers of instrumentation tools/plugins. The topic of linking methods is mentioned in other issues:

• Standardize on dynamic library symbols published by libsystrace #4 - early discussion of exposed plugin interface
• [RFC] refactor how tool is written #45 - discusses no_std vs stdlib and linking issues
• This topic is also related to the eventual support of static binaries as guests/tracees (support static binaries #28), which will require weaning off of LD_PRELOAD. (See @wangbj's xsdl effort.)

We've found ourselves constrained by the Rust toolchain, and in particular by a bug with MUSL and the --crate-type=cdylib compilation mode. For this issue, we focus on plugins written in Rust. Below is a summary of several strategies and what their respective blockers are. (Please update this directly in addition to commenting below.)

• --crate-type=cdylib: aforementioned bug
• --crate-type=dylib: undocumented ABI
• --crate-type=staticlib: may work, but needs some post-processing or a custom loader method (i.e. not LD_PRELOAD)
• --crate-type=bin: similar to staticlib, but we could have a single entrypoint, that, when called, provides some information about the small number of other function addresses of interest within the plugin. The binary would need to contain position-independent code however, similar to the -static-pie C compiler flag.

The idea with a static binary or static library would be to avoid any dependence from the plugin on any symbols in the guest, and provide our own means for the instrumentor (patcher, trampolines) to find the addresses needed within the plugin once it is injected into memory.

[rrnewton]

Creating a libcsupport milestone for all issues that are related to (or must come after) libc support. If we can make do with no_std, this can potentially be deferred a long while. For now, the goal is to document what the blocking issues are and convince ourselves we could solve it if we needed to.

[wangbj]

I find out a hack-ish way to generate freestanding libTOOL.so by using crate-type=staticlib thanks to your insight. it requires some post-processing but seems a lot easier than all the alternatives. I wish there's a cargo build post command, like mentioned in rust-lang/cargo#545 almost five years ago, certain things move fairly slowly in rust than I would have expected. Wonder if it would be better if there's a bigger name behind rust (like go).

[wangbj]

I created a simple tool for this issue: https://github.com/wangbj/rust-staticlib-linker

[wangbj]

@rrnewton I don't think it is a good idea anymore (by using rust-staticlib-linker). So we have managed to create a freestanding libTOOL.so, and we have rust libstd code in it, rust (libstd) loves thread local storage (TLS), it pretty much use it anywhere when it is possible, mainly to get rid of mutable global variables. However, TLS is implemented in libc; because our libTOOL.so statically linked musl libc code, and the traced application (being LD_PRELOAD-ed) is likely linked against glibc, they have totally different TLS implementations! Getting TLS offset is basically something like:

• get thread TCB
• get tls_offset from TCB

function pthread_self() is used for the first bulletin, in glibc, it is something like

mov    %fs:0x10,%rax

, while in musl:

mov    %fs:0x0,%rax

I even tried to patch libTOOL.so 's __tls_get_addr, to use the same one as ld-linux.so, but the app would crash/hang elsewhere, here is a snippet (gdb) where it hung at __lll_lock_wait_private():

#0  __lll_lock_wait_private () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:95
#1  0x00007ffff76684a1 in tcache_init () at malloc.c:2986
#2  0x00007ffff766c306 in tcache_init () at malloc.c:2983
#3  __libc_calloc (n=n@entry=1, elem_size=elem_size@entry=32) at malloc.c:3402
#4  0x00007ffff76157c2 in __cxa_thread_atexit_impl (func=0x7ffff7a846c0 <std::thread::local::fast::destroy_value>, obj=0x7ffff7fb37e0, 
    dso_symbol=0x7ffff7dd10b0 <_rust_extern_with_linkage___dso_handle>) at cxa_thread_atexit_impl.c:106
#5  0x00007ffff7a8f91a in std::sys::unix::fast_thread_local::register_dtor () at src/libstd/sys/unix/fast_thread_local.rs:28
#6  std::thread::local::fast::Key<T>::register_dtor () at src/libstd/thread/local.rs:386
#7  std::thread::local::fast::Key<T>::get () at src/libstd/thread/local.rs:377
#8  std::io::stdio::LOCAL_STDERR::__getit () at src/libstd/thread/local.rs:185
#9  std::thread::local::LocalKey<T>::try_with () at src/libstd/thread/local.rs:296
#10 std::io::stdio::print_to () at src/libstd/io/stdio.rs:780
#11 std::io::stdio::_eprint () at src/libstd/io/stdio.rs:811
#12 0x00007ffff79dcba3 in captured_syscall (p=0x7ffff7fee000, t=0x7fffffffd520, no=12, a0=0, a1=140737347574848, a2=131696, a3=-4096, a4=2, 
    a5=0) at examples/echo/src/entry.rs:41
#13 0x00007ffff79dbea4 in syscall_hook (info=0x7fffffffd5e0) at examples/echo/src/ffi.rs:75
#14 0x00007ffff79db16c in _syscall_hook_trampoline () at ../../trampoline/trampoline.S:80
#15 0x000000000000000c in ?? ()
#16 0x0000000000000000 in ?? ()

So we cannot really make libTOOL.so totally isolated from other shared libraries, because of TLS and threading; If we cannot use threads and TLS, I don't think there're much benefits compare to no_std.

[wangbj]

we now use dlmopen to get around of this issue, changes have been pushed into mater.

[rrnewton]

That's an amazing feat! Rare are two libc's coexisting in one process!