feat: minimal tombstone integration #4933

supervacuus · 2025-11-25T09:44:38Z

📜 Description

As discussed last week with @markushi, the process will be to make minimal atomic changes in each PR and merge directly to main rather than accumulating on an uber feature branch. This allows for easier review/feedback/corrections, and we can already test a subset of the entire feature "in the field".

The first minimal change is a basic tombstone integration:

exposes internal option to enable/disable (disabled by default)
the integration will only ever be enabled if the runtime system is at least Android 12
the current setup suggests an operation where users will have to disable NDK support (or only depend on sentry-android-core) or get two reports for the same crash
only retrieves the most current tombstone (the current implementation is not entirely correct: we should either remove any remaining ApplicationExitInfo entries with REASON_CRASH_NATIVE or report them too, including the option for the latter; I left this out for minimal interface exposure in the first step, but either variant is easy to add to this PR or later)
adds a basic tombstone parser/decoder and accompanying snapshot test
introduces the following dependencies:
- runtime: protobuf-javalite (the entire features adds ca. 75KiB to the Android sample release APK)
- build: the protobuf plugin and the protoc compiler to automate protocol updates

Open Issues:

While the protobuf runtime is relatively small, there is still the possibility of conflicting with client-side protobuf versions (major versions often introduce quite severe breakage, but I haven't tested this yet, only reviewed change logs).
finding clarity in how to proceed with older tombstones (ignore or handling similarly to ANRv2)
decision if this minimal setup already makes sense to release as an internal API
add ManifestMetadataReader to configure conveniently? (or not since the corresponding options are only internal?)
No changes yet to the UI (there are quite a few aspects that would make these stack-traces more readable, from my pov this is currently out of scope, but i am writing stuff down)

💡 Motivation and Context

First sensible release step for #3295

Part of https://linear.app/getsentry/project/tombstone-support-android-0024cb6e3ffd/

💚 How did you test it?

Added a basic parser snapshot test for a serialized tombstone protobuf.
Manual testing.

📝 Checklist

I added GH Issue ID & Linear ID
I added tests to verify the changes.
No new PII added or SDK only sends newly added PII if sendDefaultPII is enabled.
I updated the docs if needed.
I updated the wizard if needed.
Review from the native team if needed.
No breaking change or entry added to the changelog.
No breaking change for hybrid SDKs or communicated to hybrid SDKs.

🔮 Next steps

Decide if we want to go forward with client-side tombstone processing. This decision should happen with this PR.
- If no, consider sending a tombstone as an attachment, similar to how we attach minidumps and let ingestion/processing deal with decoding (I can prototype this in parallel, since I think we can do most of it in relay, as a first step)
- If yes, decide on adding the protobuf runtime library as a dependency to sentry-android-core (or shade/relocate, or implement our own decoder, given that this is a stable format which only requires a subset of protobuf).
  - Depending on feedback, adaptation to this minimal change (primarily handling older tombstones).
  - Otherwise, integrate an EventProcessor that merges crash events from sentry-native with tombstones.

#skip-changelog (for now)

…ernal)

supervacuus · 2025-11-25T09:54:53Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+    private void reportAsSentryEvent(ApplicationExitInfo exitInfo) {
+      SentryEvent event;
+      try {
+        TombstoneParser parser = new TombstoneParser(exitInfo.getTraceInputStream());


Agreed, but also left it open, since I don't know yet how the general propagation of errors is handled in integration runs (top-level handler? silent/logging discard? ...?).

We usually use a wide try/catch and silently swallow those errors in combination with logging the error using options.getLogger()

supervacuus · 2025-11-25T09:50:22Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+          latestTombstone = applicationExitInfo;
+          // remove it, so it's not reported twice
+          // TODO: if we fail after this, we effectively lost the ApplicationExitInfo (maybe only remove after we reported it)
+          exitInfos.remove(applicationExitInfo);


This is intentionally marked as TODO, so we can discuss handling history in this PR.

Yeah good point, for ANRs we write a timestamp to disk (impl) to ensure we don't double report on the next app launch. @romtsn what do you think about this? I guess what's to discuss here:

How many days back should we consider crashes?
the 90 days threshold sounds good to me

Should we report them all?
In my opinion yes. TBD: How to deal with persisted scope data for enrichment?

How to deal with unprocessable tombstones?
In my opinion we should skip them, otherwise the processing will get stuck until the crash is out of the 90 day window. TBD: should we report those as dropped events?

cursor · 2025-11-25T09:49:13Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+    private void reportAsSentryEvent(ApplicationExitInfo exitInfo) {
+      SentryEvent event;
+      try {
+        TombstoneParser parser = new TombstoneParser(exitInfo.getTraceInputStream());


Bug: Missing null check for trace input stream

ApplicationExitInfo.getTraceInputStream() can return null according to Android documentation (and as handled correctly in AnrV2Integration.parseThreadDump()). When null is passed to TombstoneParser, the subsequent TombstoneProtos.Tombstone.parseFrom(null) call in parse() will throw a NullPointerException. This exception isn't caught by the IOException catch block, causing the background TombstoneProcessor thread to crash silently without reporting the tombstone.

See: #4933 (comment)

supervacuus · 2025-11-26T15:03:56Z

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

    final Class<?> sentryNdkClass = loadClass.loadClass(SENTRY_NDK_CLASS_NAME, options.getLogger());
    options.addIntegration(new NdkIntegration(sentryNdkClass));

+    if (buildInfoProvider.getSdkInfoVersion() >= Build.VERSION_CODES.R) {


Not entirely sure about this one:

not sure if we want to add the integration conditionally here, or instead only run it conditionally (with a log that explains that the integration isn't really running)

independent of where we check this: while it is true that REASON_CRASH_NATIVE is available with R, the tombstone will not be available before S.

We can ignore the latter, since the API is available with R and we have to handle a null InputStream anyway.

As without tombstones the reported events won't be any useful I suggest to check against S instead, >= Build.VERSION_CODES.S

cursor · 2025-11-26T16:12:52Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+      SentryEvent event;
+      try {
+        TombstoneParser parser = new TombstoneParser(exitInfo.getTraceInputStream());
+        event = parser.parse();


Bug: InputStream from getTraceInputStream is never closed

The InputStream returned by exitInfo.getTraceInputStream() is passed to TombstoneParser but never closed. The existing AnrV2Integration properly uses try-with-resources to close this stream (line 306). The tombstone stream is stored in TombstoneParser.tombstoneStream and used by parseFrom(), but neither the parser nor the caller closes it afterward. This resource leak could exhaust file descriptors if the integration processes tombstones repeatedly.

cursor · 2025-11-26T18:41:14Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+        event = parser.parse();
+        event.setTimestamp(DateUtils.getDateTime(exitInfo.getTimestamp()));
+      } catch (IOException e) {
+        throw new RuntimeException(e);


Bug: Uncaught exception crashes background thread silently

When IOException occurs during tombstone parsing, wrapping it in RuntimeException and throwing terminates the background thread without logging. Since this runs in an ExecutorService submitted task, the exception is swallowed and never reported. The try-catch at line 77 only catches submission failures, not execution failures. This makes tombstone parsing failures invisible, preventing debugging and potentially losing crash reports.

cursor · 2025-11-26T18:41:14Z

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

    final Class<?> sentryNdkClass = loadClass.loadClass(SENTRY_NDK_CLASS_NAME, options.getLogger());
    options.addIntegration(new NdkIntegration(sentryNdkClass));

+    if (buildInfoProvider.getSdkInfoVersion() >= Build.VERSION_CODES.R) {


Bug: Wrong Android API level check for tombstone support

The integration checks for Build.VERSION_CODES.R (Android 11, API 30), but according to the author's own PR comment and Android documentation, tombstones via getTraceInputStream() are only available from Build.VERSION_CODES.S (Android 12, API 31). On Android R, getTraceInputStream() returns null for native crashes, which combined with the missing null check in the parser, causes the integration to fail on devices running Android 11.

supervacuus · 2025-11-26T21:08:08Z

@markushi, I just realized I cannot omit having a separate "tombstone" marker, even if I report all events, without repeating them. I mean, this was clear to me, but, in addition to it being a must for this PR already, unlike the ANR marker, I must also align it with crashedLastRun.

I wonder if it would make the most sense to do it similarly to ANR by using a TombstoneHint to ensure writing the marker timestamp at the correct life-cycle stage of the event, but introduce a new interface analog to AbnormalExit called CrashExit, so that the hint can take a different route (since it isn't truly an abnormal exit and should affect similar paths to the Native SDK crash marker).

The biggest issue with that approach is that the crashedLastRun is handled entirely in EnvelopeCache rather than AndroidEnvelopeCache, whereas ApplicationExitInfo hint/marker handling happens in the latter.

The PR still makes sense for a first review from you (since, if the general direction makes sense and you have todos not in my list, I can also add a test for the integration itself). Still, I would appreciate a short sync on how to align these execution paths (maybe I don't need to align tombstones with crashedLastRun and can abuse AbnormalExit for the same outcome, though it feels like a significant shortcut, even for minimal internal exposure, which is not at all what I aimed for).

I can convert the PR back to a draft if you prefer.

markushi

Left a few comments - great work so far!

markushi · 2025-11-27T07:07:26Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+import java.util.Locale;
+import java.util.Map;
+
+public class TombstoneParser {


If we don't want classes to be consumed outside our SDK we usually mark them as Internal

Suggested change

public class TombstoneParser {

@ApiStatus.Internal

public class TombstoneParser {

Yeah, I already have a change in progress that annotates this more strictly.

markushi · 2025-11-27T07:10:43Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+  private final InputStream tombstoneStream;
+  private final Map<String, String> excTypeValueMap = new HashMap<>();
+
+  public TombstoneParser(InputStream tombstoneStream) {


We usually are very keen on marking as much as possible as final as well as annotating any non-primitive method parameters as @Nullable or @NotNull

Suggested change

public TombstoneParser(InputStream tombstoneStream) {

public TombstoneParser(final @NotNull InputStream tombstoneStream) {

markushi · 2025-11-27T07:13:30Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+import java.util.Locale;
+import java.util.Map;
+
+public class TombstoneParser {


I guess there are different ways to deal with closing the input stream at some point, a pattern we've used in the past would be to implement the Closable interface and then initiate the class using a try-with-resources wrapping call.

Suggested change

public class TombstoneParser {

public class TombstoneParser implements Closable {

Yeah, this makes sense.

I haven't been closing anything in the integration yet because I wasn't sure about the semantics of the trace InputStream. The API docs mention that the system ring buffer evicts tombstones, which made me a bit wary RE: statefulness; however, after reading the code, it turns out that only means the returned InputStream could be null (which it can be for many reasons actually; curious choice to highlight the tombstone rotation).

Each returned trace stream is wrapped as an AutoCloseInputStream on a fresh parcel fd

The tombstones the app searches through were retrieved by system_server via a file watcher on /data/tombstones

and the rotation mentioned in the API docs only happens in the CrashQueue of tombstoned via unlinkat and linkat.

All of the above operate on the same set of files (shared between system daemons and all apps, where binder delivery per app/user is decided in the system_server module linked above), however, we still operate in classic UNIX territory, aka unlink removes the path name but not the inode if there are open fds.

markushi · 2025-11-27T07:19:50Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+      thread.setStacktrace(stacktrace);
+      if (tombstone.getTid() == threadEntry.getValue().getId()) {
+        thread.setCrashed(true);
+        // even though we refer to the thread_id from the exception,


I'd say that's the perfect level of code comments I'd like to see, nice!

markushi · 2025-11-27T07:24:13Z

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

+    // https://android.googlesource.com/platform/system/unwinding/+/refs/heads/main/libunwindstack/Regs.cpp#175
+    // prevent "processing" from doing it again.
+    unknown.put("instruction_addr_adjustment", "none");
+    stacktrace.setUnknown(unknown);


unknown is mainly used for hybrid SDKs or when the Java SDK is used as a sending mechanism to ensure no data is lost due to parsing. In this case I'd favor extending the SentryStackTrace class with a private @Nullable String instructionAddressAdjustment field instead.

markushi · 2025-11-27T07:32:07Z

sentry-android-core/src/main/java/io/sentry/android/core/AndroidOptionsInitializer.java

    final Class<?> sentryNdkClass = loadClass.loadClass(SENTRY_NDK_CLASS_NAME, options.getLogger());
    options.addIntegration(new NdkIntegration(sentryNdkClass));

+    if (buildInfoProvider.getSdkInfoVersion() >= Build.VERSION_CODES.R) {


As without tombstones the reported events won't be any useful I suggest to check against S instead, >= Build.VERSION_CODES.S

markushi · 2025-11-27T07:57:55Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+          latestTombstone = applicationExitInfo;
+          // remove it, so it's not reported twice
+          // TODO: if we fail after this, we effectively lost the ApplicationExitInfo (maybe only remove after we reported it)
+          exitInfos.remove(applicationExitInfo);


Yeah good point, for ANRs we write a timestamp to disk (impl) to ensure we don't double report on the next app launch. @romtsn what do you think about this? I guess what's to discuss here:

How many days back should we consider crashes?
the 90 days threshold sounds good to me

Should we report them all?
In my opinion yes. TBD: How to deal with persisted scope data for enrichment?

How to deal with unprocessable tombstones?
In my opinion we should skip them, otherwise the processing will get stuck until the crash is out of the 90 day window. TBD: should we report those as dropped events?

markushi · 2025-11-27T08:01:32Z

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

+    private void reportAsSentryEvent(ApplicationExitInfo exitInfo) {
+      SentryEvent event;
+      try {
+        TombstoneParser parser = new TombstoneParser(exitInfo.getTraceInputStream());


We usually use a wide try/catch and silently swallow those errors in combination with logging the error using options.getLogger()

markushi · 2025-11-27T08:03:51Z

sentry-android-core/src/main/proto/io/sentry/android/core/internal/tombstone/tombstone.proto

@@ -0,0 +1,204 @@
+// Added and adapted from: https://android.googlesource.com/platform/system/core/+/refs/heads/main/debuggerd/proto/tombstone.proto


We should also add some license attribution here, e.g. like we did here.

markushi · 2025-11-27T08:06:57Z

gradle/libs.versions.toml

 otel-semconv = { module = "io.opentelemetry.semconv:opentelemetry-semconv", version.ref = "otelSemanticConventions" }
 otel-semconv-incubating = { module = "io.opentelemetry.semconv:opentelemetry-semconv-incubating", version.ref = "otelSemanticConventionsAlpha" }
 p6spy = { module = "p6spy:p6spy", version = "3.9.1" }
+protobuf-javalite = { module = "com.google.protobuf:protobuf-javalite", version.ref = "protobuf"}


@romtsn Not sure if you followed the conversations, but as you can see here, protobuf requires a runtime dependency. It will have an impact of around 10kb. IMHO fine for now, we should still check how stable this library is to avoid and consumer version mismatch issues.

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java

feat: minimal tombstone integration (disabled by default, options int…

ba0c7db

…ernal)

supervacuus requested review from adinauer, lcian, markushi, romtsn and stefanosiano as code owners November 25, 2025 09:44

sentry bot reviewed Nov 25, 2025

View reviewed changes

cursor bot reviewed Nov 25, 2025

View reviewed changes

supervacuus commented Nov 26, 2025

View reviewed changes

redo a seemingly missed spotlessApply

9933e47

cursor bot reviewed Nov 26, 2025

View reviewed changes

supervacuus added 2 commits November 26, 2025 19:36

use non-plural option + mark integration also as internal + update API

10b1e94

Merge branch 'main' into feat/tombstone_integration

f12b658

cursor bot reviewed Nov 26, 2025

View reviewed changes

markushi reviewed Nov 27, 2025

View reviewed changes

init size exceptions since we know we only need one

4b6e1fb

Co-authored-by: Markus Hintersteiner <m.hintersteiner@gmail.com>

sentry bot reviewed Nov 27, 2025

View reviewed changes

...ry-android-core/src/main/java/io/sentry/android/core/internal/tombstone/TombstoneParser.java Show resolved Hide resolved

sentry-android-core/src/main/java/io/sentry/android/core/TombstoneIntegration.java Show resolved Hide resolved

	public class TombstoneParser {
	@ApiStatus.Internal
	public class TombstoneParser {

	public TombstoneParser(InputStream tombstoneStream) {
	public TombstoneParser(final @NotNull InputStream tombstoneStream) {

	public class TombstoneParser {
	public class TombstoneParser implements Closable {

		@@ -0,0 +1,204 @@
		// Added and adapted from: https://android.googlesource.com/platform/system/core/+/refs/heads/main/debuggerd/proto/tombstone.proto

Uh oh!

feat: minimal tombstone integration #4933

Are you sure you want to change the base?

feat: minimal tombstone integration #4933

Uh oh!

Conversation

supervacuus commented Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

📜 Description

💡 Motivation and Context

💚 How did you test it?

📝 Checklist

🔮 Next steps

Uh oh!

This comment was marked as outdated.

Uh oh!

supervacuus Nov 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as outdated.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Nov 25, 2025

Choose a reason for hiding this comment

Bug: Missing null check for trace input stream

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor bot Nov 26, 2025

Choose a reason for hiding this comment

Bug: InputStream from getTraceInputStream is never closed

Uh oh!

cursor bot Nov 26, 2025

Choose a reason for hiding this comment

Bug: Uncaught exception crashes background thread silently

Uh oh!

cursor bot Nov 26, 2025

Choose a reason for hiding this comment

Bug: Wrong Android API level check for tombstone support

Uh oh!

supervacuus commented Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markushi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

supervacuus commented Nov 25, 2025 •

edited

Loading

supervacuus Nov 25, 2025 •

edited

Loading

supervacuus commented Nov 26, 2025 •

edited

Loading