AsyncBencher::iter and friends with the tokio runtime uses Runtime::block_on to "do work" which will cause significant slowdown in multi-threaded runtime

[jhgg]

Consider the following benchmark code:

use criterion::Criterion;
use criterion::{criterion_group, criterion_main};
use tokio::{
    runtime::Builder,
    sync::{
        mpsc::{channel, Receiver},
        oneshot,
    },
};

async fn ping_pong(mut rx: Receiver<oneshot::Sender<()>>) {
    while let Some(sender) = rx.recv().await {
        sender.send(()).ok();
    }
}

fn runtimes(c: &mut Criterion) {
    c.bench_function("current thread", |b| {
        let rt = Builder::new_current_thread().build().unwrap();
        let (tx, rx) = channel(1);
        rt.spawn(ping_pong(rx));

        b.to_async(&rt).iter(|| async {
            let (a, b) = oneshot::channel();

            tx.send(a).await.unwrap();
            b.await.unwrap();
        })
    });

    c.bench_function("multi thread", |b| {
        let rt = Builder::new_multi_thread()
            .worker_threads(4)
            .build()
            .unwrap();
        let (tx, rx) = channel(1);
        rt.spawn(ping_pong(rx));

        b.to_async(&rt).iter(|| async {
            let (a, b) = oneshot::channel();

            tx.send(a).await.unwrap();
            b.await.unwrap();
        })
    });
}

criterion_group!(benches, runtimes);
criterion_main!(benches);

We can notice a significant change in benchmark performance when running these two identical benchmarks where only the runtime differs.

     Running benches/rt.rs (target/release/deps/rt-b193580270194018)
current thread          time:   [186.69 ns 187.38 ns 188.26 ns]
Found 4 outliers among 100 measurements (4.00%)
  4 (4.00%) high severe

multi thread            time:   [34.798 µs 35.261 µs 35.804 µs]
Found 8 outliers among 100 measurements (8.00%)
  2 (2.00%) low mild
  2 (2.00%) high mild
  4 (4.00%) high severe

This caveat is noted in tokio's documentation

Non-worker future

Note that the future required by this function does not run as a worker. The expectation is that other tasks are spawned by the future here. Awaiting on other futures from the future provided here will not perform as fast as those spawned as workers.

However, I do not know if this caveat is apparent to users of the criterion crate.