Skip to content

Conversation

@szokeasaurusrex
Copy link
Member

@szokeasaurusrex szokeasaurusrex commented Jan 14, 2026

Description

This PR does three important things, which are all interdependent, so I think it makes sense to do it all in a single PR.

(1) Making HubSwitchGuard !Send

This PR makes HubSwitchGuard !Send by adding PhantomData<MutexGuard<'static, ()>> while keeping it Sync. The type system now prevents the guard from being moved across threads at compile time.

This change is important because HubSwitchGuard manages thread-local hub state, but was previously Send, allowing it to be moved to another thread. When dropped on the wrong thread, it could corrupt that thread's hub state instead of restoring the original thread. This change resolves #943.

Tests related to this change: !Send is enforced by the compiler, so there are no additional tests here.

(2) A new stack for spans to manage HubSwitchGuards

As HubSwitchGuard is now !Send, we needed a new way to manage the HubSwitchGuard associated with a given span, in the tracing integration. Previously, we just had the guards live directly on the span via the SentrySpanData type, but this no longer works, since spans need to be Send. So, we now declare a thread-local mapping from span IDs to HubSwitchGuards. The guards are stored in a stack, as each span will have one guard per span entry (spans can be entered multiple times, even on the same thread, and without the stack, we would restore the original hub too early). We drop the guards on span exit in LIFO order.

Tests related to this change: span_reentrancy.rs contains tests to validate correct span tree structure when re-entering the same span multiple times. A previous iteration of this PR only allowed a single guard per span; the test failed against this implementation because it produces an incorrect span structure (two transactions instead of one). The test only passes with the guard stack.

(3) Forking the Hub on each span (re-)entry

Change (2) is insufficient to make the span_reentrancy.rs test pass because with only that change, there is still the fundamental problem that each span does not get its own Hub to manage state. Thus, in that test, I believe we were actually only ever using one hub, because there was no place we were forking the hub. So, on span exit, we prematurely set the parent span on the hub.

Forking the hub ensures proper isolation, so the span gets set back at the right time (I also suspect we don't need to manually set it back, but I am unsure).

This change resolves #946.

Tests related to this change: span_cross_thread.rs ensures that entering the same span in multiple threads produces the correct span tree. Basically, it is a reproduction of #946. span_reentrancy.rs also only passes with this change.

Issues

@linear
Copy link

linear bot commented Jan 14, 2026

@szokeasaurusrex szokeasaurusrex force-pushed the szokeasaurusrex/hubswitchguard branch from 21b407b to 1dea844 Compare January 14, 2026 12:28
@szokeasaurusrex szokeasaurusrex changed the title fix(core): Make HubSwitchGuard !Send to prevent thread corruption fix!(core): Make HubSwitchGuard !Send to prevent thread corruption Jan 14, 2026
@szokeasaurusrex szokeasaurusrex force-pushed the szokeasaurusrex/hubswitchguard branch from e78483a to 8abf004 Compare January 14, 2026 12:54
@szokeasaurusrex szokeasaurusrex requested a review from lcian January 26, 2026 16:19
@szokeasaurusrex szokeasaurusrex force-pushed the szokeasaurusrex/hubswitchguard branch from 8abf004 to 1255c8a Compare January 28, 2026 12:36
@szokeasaurusrex szokeasaurusrex marked this pull request as ready for review January 28, 2026 14:23
Copy link

@cursor cursor bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Bugbot Autofix is OFF. To automatically fix reported issues with Cloud Agents, enable Autofix in the Cursor dashboard.

lcian
lcian previously approved these changes Feb 2, 2026
Copy link
Member

@lcian lcian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

@szokeasaurusrex szokeasaurusrex force-pushed the szokeasaurusrex/hubswitchguard branch from 0976b79 to e4ddd01 Compare February 4, 2026 12:57
@szokeasaurusrex szokeasaurusrex marked this pull request as draft February 4, 2026 15:34
@szokeasaurusrex
Copy link
Member Author

This current implementation is still flawed; if the same span is re-entered in the same thread (possible in async contexts), the second entry overwrites the original HubSwitchGuard, corrupting the behavior. I am working on a fix

@lcian
Copy link
Member

lcian commented Feb 4, 2026

@szokeasaurusrex it rewrites the original guard with the same one I think, so what's the issue there?

@lcian
Copy link
Member

lcian commented Feb 4, 2026

I've tested some scenarios manually and didn't find any issues. I assume you're also testing manully.
Would be nice to add these as automated tests as well.

@szokeasaurusrex
Copy link
Member Author

szokeasaurusrex commented Feb 4, 2026

@lcian Codex AI agent identified the potential issue here as I was working on #946. I have a test locally which reproduces the behavior. Will commit it with a more detailed explanation of the problem tomorrow 👍

I suppose it's possible that Codex is wrong and the local test is doing something which is not supposed to ever happen (the scenario is admittedly a bit contrived), but in any case, I think it's worth it to investigate properly. So, that is what I'm doing now. I will let you know if I need any assistance

szokeasaurusrex and others added 5 commits February 5, 2026 14:58
HubSwitchGuard manages thread-local hub state but was Send, allowing it
to be moved to another thread. When dropped on the wrong thread, it would
corrupt that thread's hub state instead of restoring the original thread.

To fix this, add PhantomData<MutexGuard<'static, ()>> to make the guard
!Send while keeping it Sync. This prevents the guard from being moved
across threads at compile time.

Additionally, refactor sentry-tracing to store guards in thread-local
storage keyed by span ID instead of in span extensions. This fixes a
related bug where multiple threads entering the same span would clobber
each other's guards.

Fixes #943
Refs RUST-130

Co-Authored-By: Claude <noreply@anthropic.com>
@szokeasaurusrex szokeasaurusrex force-pushed the szokeasaurusrex/hubswitchguard branch from e4ddd01 to 6169b08 Compare February 5, 2026 14:02
@szokeasaurusrex szokeasaurusrex changed the base branch from master to szokeasaurusrex/envelope-into_items February 5, 2026 14:02
Copy link
Member Author

@szokeasaurusrex szokeasaurusrex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have made some pretty substantial changes here. Would appreciate a re-review.

Please also see the updated description 🙏

{
let _enter_b = span_b.enter();
}
}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lcian This reproduces the bug I was talking about. With the original implementation, span_a and span_b ended up as two separate transactions because dropping _reenter_a restored the original hub. There were two underlying issues here that were causing this behavior; I fixed both of them here.

The first fix was that I changed the SPAN_GUARDS so that it would not just have a single guard per span, but rather, to have a stack of guards that are pushed on each entry and popped on each exit. This way, we only restored the original hub on the final exit from the span.

That first fix was only part of the problem, though. To get the span structure correct here, I also needed to fix #946. For that, we needed to be forking the hub, as you suggested last week. Otherwise, if we just use the hub on the span directly, we end up not actually getting proper scope isolation.

Comment on lines 387 to 389
data.hub.configure_scope(|scope| {
scope.set_span(data.parent_sentry_span.clone());
});
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that we are forking the hub, I actually don't think we need to be setting the parent span on the hub here; the hub we restore should already have the parent span set (unless some other code intentionally changed the span on that hub, which we probably should respect).

Suggested change
data.hub.configure_scope(|scope| {
scope.set_span(data.parent_sentry_span.clone());
});

What do you think @lcian? Am I missing something?

@szokeasaurusrex szokeasaurusrex dismissed lcian’s stale review February 6, 2026 10:12

PR is substantially change, should be re-reviewed

@szokeasaurusrex szokeasaurusrex marked this pull request as ready for review February 6, 2026 10:12
@szokeasaurusrex szokeasaurusrex requested a review from lcian February 6, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HubSwitchGuard should not be Send

2 participants