Instrumented executable runs 170 times slower.

[alex-smartptr]

This is a very nebulous problem description I'm afraid. My test program consistently runs in 32ms without instrumentation, and 5500ms with funtrace instrumentation.

The timing is very consistent. I doesn't matter which compiler or wrapper I use, the program always runs in 5500ms.

From what I can tell using the traces that it produces, the program pauses for hundreds of milliseconds on pthread_join. I'm not sure where to start, so any suggestions are welcome.

[yosefk]

what does this test program do, in rough terms? 32ms sounds like a microbenchmark - do you see this slowdown in a larger program? thread creation and exit in particular is slower under funtrace, eg it allocates/frees a trace buffer, and takes a lock - the latter bit is mentioned under limitations:

"Thread creation/exit and saving a trace snapshot take the same lock - this can slow things down; hopefully not too badly since saving a snapshot is pretty fast, and creating lots of threads at runtime (rather than reusing from a thread pool) should be rare"

[alex-smartptr]

what does this test program do, in rough terms?

Good question. I am currently investigating that. It definitely uses a scatter/gather pattern - starting some threads to parallelise some work, and then waiting for them all to complete. It does that a couple of times, possibly many times, over the course of the run.

I'm still investigating what it actually does - I've been handed this run as an exemplar, and asked to make it go faster. I'm not previously familiar with the code.

I've simply added a call to funtrace_pause_and_write_current_snapshot(); right at the end of the run. It's not competing with thread create/join during the actual work.

It feels to me like something is timing out, probably the same thing multiple times. Thanks for the suggestion about thread creation - I'll keep an eye on that. Once I have a better understanding of what the code is doing, I may be able to offer more information to help identify the slow down.

[yosefk]

If the purpose of those threads is to be worker threads for parallelization, you could save the overhead of threads starting and finishing using a thread pool (and it should speed things up regardless of funtrace) - via TBB or via C++17 parallel algorithms, something like this:

#include <algorithm>
#include <execution>
#include <vector>
#include <iostream>

int main() {
    std::vector<int> vec = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10};

    std::for_each(std::execution::par, vec.begin(), vec.end(), [](int& x) {
        std::cout << x << " processed by thread " << std::this_thread::get_id() << "\n";
    });

    return 0;
} 

If you want to figure out what the program is doing using funtrace, you might want a bigger buffer and a bigger thread GC timeout - see $FUNTRACE_LOG_BUF_SIZE and $FUNTRACE_GC_MAX_AGE_MS in README

[alex-smartptr]

I hacked in a toy thread pool that works with the code's thread usage, and magically it now runs in 26ms. So that's clearly the issue. Thanks for the suggestion.

I have not tried fiddling with the env. vars. you mentioned yet. I'll keep this issue alive until I can give that a try.