Benchmark SDK startup

[bruno-garcia]

A user reported that while benchmarking their app's startup, the Sentry SDK had significant overhead. Unfortunately I don't have more details than that.

We should benchmark the init call with default/common settings and verify if anything can be made async, or otherwise improved to reduce the overhead.

A clear communication of what's happening during init can help set expectations of overhead. Including some example benchmarks.

[supervacuus]

Before starting with a benchmark, I can tell you immediately that the most significant cost (i.e., defining the init-time) when using the crashpad backend is the start of the crashpad_handler and its IPC bootstrap.

On my M1, this takes 75ms on average and can peak at 100+ms, which isn't great but happens 100% outside the SDK. On the same machine, initializing the SDK with the breakpad backend (which doesn't start another process) rarely takes longer than 10ms end-to-end (and averages to 5ms). I expect the relative init times to be similar on other platforms.

So, what is the expected upper bound here? Should we target below 5ms? Is breakpad a sensible alternative for users targeting init-times below 75ms?

While we can async the handler's start, this would require a callback, at least when the handler start was unsuccessful, because it can't report any crashes through the application lifetime, which can be acceptable. Still, it should be a decision the client can make.

Otherwise, if that makes sense, I am all for adding benchmarks or improving the documentation towards expected runtimes.

[bruno-garcia]

Before starting with a benchmark, I can tell you immediately that the most significant cost (i.e., defining the init-time) when using the crashpad backend is the start of the crashpad_handler and its IPC bootstrap.

While this is useful context, I think it doesn't replace the need for a benchmark. In fact, it raises a good point that we'd need benchmarks with both breakpad and crashpad backends.

So, what is the expected upper bound here? Should we target below 5ms? Is breakpad a sensible alternative for users targeting init-times below 75ms?

What is acceptable will vary from app/customer to app/customer. I'd say we publish benchmarks, and go from there. Doesn't mean we need to improve things immediately, but at least we're being transparent about things.

While we can async the handler's start, this would require a callback, at least when the handler start was unsuccessful, because it can't report any crashes through the application lifetime, which can be acceptable. Still, it should be a decision the client can make.

Yeah if it's an opt-in, I imagine some users might rather the more complex init with a callback, and the risk of missing some crashes until the SDK initializes. But we don't need to discuss this just yet. Lets take a look at the benchmarks and then see what are the biggest opportuntiies

[supervacuus]

While this is useful context, I think it doesn't replace the need for a benchmark. In fact, it raises a good point that we'd need benchmarks with both breakpad and crashpad backends.

I fully agree. The context wasn't meant as a counter to introduce benchmarks into the Native SDK codebase but as a framing for creating such a benchmark and how to address users.

Regarding the benchmark itself:

• we are focusing on sentry_init() for the time being?
• do we want to record only the init's runtime or its components?
• would the benchmark matrix look like:
  • crashpad, breakpad, inproc
  • Windows, macOS, Linux, Android(?)
  • only release builds?
  • Should the benchmark run in a (virtual) CI environment or on local machines?
• related to all of the above: is the primary concern to inform users or to target potential improvements? Or is the latter meant as a follow-up?

[bruno-garcia]

I'd leave the details to you. I imagine implementing all things you listed could be very time consuming so I'd only suggest trying to take a simpler route. Something quick that gives good insight might take us far, and we can always revisit this and go deeper if needed.