i#5383: macOS a64 client threads and private TLS #7300

ndrewh · 2025-02-24T20:10:17Z

Adds macOS ARM64 support for client threads and private TLS (under -private_loader -- though we don't implement a full private loader yet).

Separates macOS x86 and A64 private TLS into separate files. The new A64 private TLS uses TLS_TYPE_SLOT in a similar manner to the linux riscv TLS implementation.
A64 handler for new_bsdthread_intercept
Fix wrong exit syscall in client_thread_run (it currently exits the entire process rather than thread)

The end result is that client threads can start and terminate properly, and multithreaded applications can also terminate without crashing. I did not test attach/detach, but I'm guessing it's still broken (there is no injector implementation anyway iiuc)

ndrewh · 2025-02-24T20:11:01Z

core/arch/aarch64/aarch64.asm

 cat_spin:
-        CALLC2(GLOBAL_REF(atomic_swap), x26, #1)
+        CALLC2(GLOBAL_REF(atomic_swap), x0, #1)


Had to change since x26 is used on line 365

OK, works out I think b/c it's in the 1st arg slot already, but it can be fragile to use the calling conv params in the args of CALL* b/c they can be clobbered by other args in arg setup.

ndrewh · 2025-02-24T20:24:56Z

core/unix/signal.c

@@ -1246,7 +1246,7 @@ signal_thread_inherit(dcontext_t *dcontext, void *clone_record)
         * FIXME: are current pending or blocked inherited?
         */
 #ifdef MACOS
-        if (record->app_thread_xsp != 0) {
+        if (record->app_thread_xsp == 0) {


I don't really understand this code but you'll get heap oob/uaf asserts if this condition is !=.

This is freeing the clone record.
See the comment in clone_record_t:

#ifdef MACOS /* XXX i#1403: once we have lower-level, earlier thread interception we can * likely switch to something closer to what we do on Linux. * This is used for bsdthread_create, where app_thread_xsp is NULL; * for vfork, app_thread_xsp is non-NULL and this is unused. */

This change != to == looks suspect. Please double check vs this pasted comment: if bsdthread_create ends up different on aarch64 please update that comment; does it need separate handling a64 vs x86.

ndrewh · 2025-02-25T18:25:31Z

Some messy stuff I'm not sure if there's a better way to do:

In dynamo_thread_init we allocate temporary TLS until os_tls_init because we will SEGV in mig_get_reply_port sometimes when acquiring locks if thread register is NULL (this occurs in client threads, which do not inherit TLS).
In dynamo_thread_exit_common we cannot use app TLS for a similar reason, because the app TLS is free'd by pthread_terminate before we intercept the call to bsdthread_terminate.

You'll get a backtrace like this if we do not maintain a valid thread register during entry and exit.

* thread #12, stop reason = EXC_BAD_ACCESS (code=1, address=0x10)
  * frame #0: 0x00000001983c24e4 libsystem_kernel.dylib`mig_get_reply_port + 24
    frame #1: 0x00000001983c5144 libsystem_kernel.dylib`semaphore_create + 52
    frame #2: 0x000000010121b59c libdynamorio.dylib`mutex_get_contended_event [inlined] ksynch_init_var(synch=0x0000000300e73ad8) at ksynch_macos.c:87:9 [opt]
    frame #3: 0x000000010121b580 libdynamorio.dylib`mutex_get_contended_event(lock=0x00000001012481c0) at ksynch_macos.c:162:14 [opt]
    frame #4: 0x000000010120cf6c libdynamorio.dylib`mutex_wait_contended_lock(lock=0x00000001012481c0, mc=0x0000000000000000) at os.c:10477:30 [opt]
    frame #5: 0x000000010109ff14 libdynamorio.dylib`d_r_mutex_lock [inlined] d_r_mutex_lock_app(lock=<unavailable>, mc=0x0000000000000000) at utils.c:884:9 [opt]
    frame #6: 0x000000010109fe88 libdynamorio.dylib`d_r_mutex_lock(lock=<unavailable>) at utils.c:897:5 [opt]
    frame #7: 0x0000000101082530 libdynamorio.dylib`dynamo_thread_init(dstack_in="", mc=0x0000000000000000, os_data=0x0000000300e73c00, client_thread=true) at dynamo.c:2290:5 [opt]
    frame #8: 0x000000010120795c libdynamorio.dylib`client_thread_run at os.c:4110:5 [opt]

derekbruening · 2025-03-01T16:57:47Z

under -private_loader -- though we don't implement a full private loader yet

Probably orthogonal to this PR: but do you think it is possible to load private copies of library on OSX? #1285 has some discussion about whether it will ever work well. Xref #7312 has some discussion of alternatives.

derekbruening

Thank you for contributing. I have mostly style comments but also some points need clarifying.

derekbruening · 2025-03-02T00:31:52Z

core/arch/aarch64/aarch64.asm

 cat_spin:
-        CALLC2(GLOBAL_REF(atomic_swap), x26, #1)
+        CALLC2(GLOBAL_REF(atomic_swap), x0, #1)


OK, works out I think b/c it's in the 1st arg slot already, but it can be fragile to use the calling conv params in the args of CALL* b/c they can be clobbered by other args in arg setup.

derekbruening · 2025-03-02T21:32:52Z

core/arch/x86_code.c

@@ -325,48 +325,6 @@ new_thread_setup(priv_mcontext_t *mc)
    ASSERT_NOT_REACHED();
 }

-#    if defined(MACOS) && defined(X86)
-/* Called from new_bsdthread_intercept for targeting a bsd thread user function.


Despite the legacy name, this file "x86_code.c" is used with asm code on all arches and not just x86. I don't think this should be moved if the only reason was the file name. If you want to rename it to asm_aux.c or something like that, that seems reasonable.

derekbruening · 2025-03-02T21:34:38Z

core/dynamo.c

@@ -85,6 +85,11 @@
 #    include "vmkuw.h"
 #endif

+#if defined(MACOS) && defined(AARCH64)
+#    include "unix/tls.h"


Generally we avoid including private headers: os_public.h and os_exports.h are what are supposed to be public.

derekbruening · 2025-03-02T21:35:28Z

core/dynamo.c

@@ -2246,6 +2257,19 @@ dynamo_thread_init(byte *dstack_in, priv_mcontext_t *mc, void *os_data,
        return SUCCESS;
    }

+/* macOS aarch64 will sometimes crash when acquiring locks if TLS is NULL */
+#if defined(MACOS) && defined(AARCH64)
+    void *tmp_tls = NULL;


Please move this inside a unix/ file and export a function to call here.

derekbruening · 2025-03-02T21:35:47Z

core/dynamo.c

@@ -2300,6 +2324,13 @@ dynamo_thread_init(byte *dstack_in, priv_mcontext_t *mc, void *os_data,
    }

    os_tls_init();
+#if defined(MACOS) && defined(AARCH64)
+    if (tmp_tls) {


Same here: let's isolate TLS details like this inside unix/

derekbruening · 2025-03-02T22:12:21Z

core/unix/os.c

+    return new_thread;
+#    else
+    /* NYI */
+    return -1;


Add ASSERT_NOT_IMPLEMENTED and up put a XXX comment with an issue number.

derekbruening · 2025-03-02T22:13:45Z

core/unix/os.c

+#    ifdef X64
+        /* Also update the pthread->fun and pthread->arg fields, since _pthread_start uses
+         * them instead of the syscall arg0 on some macOS versions */
+        ASSERT(sys_param(dcontext, 3) &&


style: explicit bools

derekbruening · 2025-03-02T22:13:53Z

core/unix/os.c

+        /* Also update the pthread->fun and pthread->arg fields, since _pthread_start uses
+         * them instead of the syscall arg0 on some macOS versions */
+        ASSERT(sys_param(dcontext, 3) &&
+               "bsdthread_create pthread argument should not be NULL");


Use ASSERT_MESSAGE

derekbruening · 2025-03-02T22:15:18Z

core/unix/os.c

+#    ifdef X64
+        /* Also update the pthread->fun and pthread->arg fields, since _pthread_start uses
+         * them instead of the syscall arg0 on some macOS versions */
+        ASSERT(sys_param(dcontext, 3) &&


I don't think we should ever assert on an app-supplied value: we want to be able to run any app. If the parameter is null, just skip this code gracefully. Use ASSERT_CURIOSITY if you want a non-fatal notification, or SYSLOG_INTERNAL_WARNING for a less visible one.

derekbruening · 2025-03-02T22:19:00Z

core/unix/signal.c

@@ -1246,7 +1246,7 @@ signal_thread_inherit(dcontext_t *dcontext, void *clone_record)
         * FIXME: are current pending or blocked inherited?
         */
 #ifdef MACOS
-        if (record->app_thread_xsp != 0) {
+        if (record->app_thread_xsp == 0) {


This is freeing the clone record.
See the comment in clone_record_t:

#ifdef MACOS /* XXX i#1403: once we have lower-level, earlier thread interception we can * likely switch to something closer to what we do on Linux. * This is used for bsdthread_create, where app_thread_xsp is NULL; * for vfork, app_thread_xsp is non-NULL and this is unused. */

This change != to == looks suspect. Please double check vs this pasted comment: if bsdthread_create ends up different on aarch64 please update that comment; does it need separate handling a64 vs x86.

ndrewh added 3 commits February 24, 2025 15:06

i#5383: macOS a64 client threads and private TLS

93bccab

fix

1bbc27a

clang-format

4539c1f

ndrewh commented Feb 24, 2025

View reviewed changes

ndrewh added 2 commits February 24, 2025 15:32

fix ifdef in dynamorio_clone_macos

a23266c

fix x86 macos build, comments

9830068

ndrewh added 2 commits February 26, 2025 12:52

clean

0d395f1

remove unrelated changes for separate PR

b6fac84

ndrewh marked this pull request as ready for review February 26, 2025 18:16

derekbruening reviewed Mar 2, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

i#5383: macOS a64 client threads and private TLS #7300

i#5383: macOS a64 client threads and private TLS #7300

ndrewh commented Feb 24, 2025 •

edited

Loading

ndrewh Feb 24, 2025

derekbruening Mar 2, 2025

ndrewh Feb 24, 2025

derekbruening Mar 2, 2025

ndrewh commented Feb 25, 2025 •

edited

Loading

derekbruening commented Mar 1, 2025

derekbruening left a comment

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

derekbruening Mar 2, 2025

i#5383: macOS a64 client threads and private TLS #7300

Are you sure you want to change the base?

i#5383: macOS a64 client threads and private TLS #7300

Conversation

ndrewh commented Feb 24, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndrewh commented Feb 25, 2025 • edited Loading

derekbruening commented Mar 1, 2025

derekbruening left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ndrewh commented Feb 24, 2025 •

edited

Loading

ndrewh commented Feb 25, 2025 •

edited

Loading