Add a succinct, "consumer grade" benchmark (or document how to run one from the current selection)

[RJVB]

Would it be possible to add a succinct benchmark where "mere users" can compare the performance benefits of mimalloc and some other common alternative allocators against the standard/system allocator, in a more or less representative workload? Or document how to run one from the available code?

An option to build against the installed allocators would be great too!

[mjp41]

Would it be possible to add a succinct benchmark where "mere users" can compare the performance benefits of mimalloc and some other common alternative allocators against the standard/system allocator, in a more or less representative workload? Or document how to run one from the available code?

This suite was design by @daanx to be used to find problems with allocators rather than a consumer facing pick you allocator.

That said, there are several benchmarks that are more representative of realistic workloads in the suite

• redis
• rocksdb
• gs
• lua
• cfrac

Running those will tell you how well those applications might perform. If they are close to your workload, then they might inform your choice.

There are other aspects of allocator performance not tested by these benchmarks:

• long term fragmentation
• massive memory footprints
• latency profiles
• memory usage over time (not just peak)

An option to build against the installed allocators would be great too!

You can specify

  ../../bench.sh --external=file.txt [set of benchmarks]

where file.txt contains lines of the form

allocator_name /path/to/allocator/library.so

This will then run with those allocators. For example, I just ran

../../bench.sh --external=alloc-16-32.txt allt -r=40

where alloc-16-32.txt

sn-16 /home/mjp41/snmalloc/release/libsnmallocshim-16.so
sn-32 /home/mjp41/snmalloc/release/libsnmallocshim-32.so

[RJVB]

Thanks, with that additional information I've been able to cobble up something for macports! (https://github.com/RJVB/macstrop/blob/678f0708981ce92f76a6b6154a75ed3bdb3eb4de/devel/mimalloc/Portfile#L61)

[RJVB]

FWIW, I see very little performance benefit in terms of timing in real-world tasks, even in a python/sqlite benchmark of operations on an in-memory database. What's more: the system allocator (Mac and Linux) seems to be the most economical in terms of total memory used, page faults and context switches (if I interpret the output from GNU time -v correctly).

Fortunately my main interest for using a malloc & family replacement has always been in hope that they'd give freed memory back to the system more aggressively and thus lead to less swap space usage and/or fragmentation. Jemalloc has an optional background thread that takes care of that; it's much less clear if and how the others do this.

[derSteFfi]

given that with #194, we increased the real-life benchmarks in the suite, the remaining issue here would be to document how to run a custom benchmark. I can imagine offering a CLI switch to pass in a custom command, or a file containing a list of commands. Assuming these commands do not need preparation or caretaking after being run, this would be rather easy to implement. I'll draft up something.