IO: reduce IO by avoiding using `write()`

[matttbe]

When trying to validate network performances on embedded system, a limited factor might be the IO. This can be avoided by avoiding copies.

There are multiple advanced zero-copy techniques these days (io_uring Zerocopy, TCP devmem, MSG_TX_RX_ZC, and probably others -- check this to get some comparisons) but they can be complex to put in place, requiring a recent Linux kernel, mostly useful for very high BW requirements, and/or requiring specific HW. (Maybe io_uring ZC might be easier to put in place, but not sure whether it is really needed with a static buffer and the need of checking exchanged data.) Yet, a first step might be to use mmap(), sendfile() and/or MSG_ZEROCOPY (not supported by all protocols) :)

(I just found this article that can help to understand why it can be interesting to do that)

[lance0]

Thanks for the detailed writeup and the NetDevConf link — that ZC comparison talk is great context.

This is on the roadmap already under "100G+ Optimization" (io_uring, MSG_ZEROCOPY, AF_XDP), but your point about embedded systems is a good reframe — it's not just about hitting higher throughput, it's about reducing CPU overhead on constrained devices where the copies themselves are the bottleneck.

I think the practical progression would be:

1. MSG_ZEROCOPY on the send path (TCP/UDP, setsockopt(SO_ZEROCOPY) + send() with MSG_ZEROCOPY flag) — lowest effort, kernel 4.14+
2. sendfile() with a pre-filled temp file — avoids userspace buffer entirely, but means we'd need to manage a scratch file
3. io_uring with fixed buffers — biggest win but biggest lift, and requires a different async runtime story (tokio-uring or glommio)

I'll label this as an enhancement. Any sense of what throughput range you're testing on the embedded side? Curious whether the copy overhead is showing up as CPU saturation or as a throughput ceiling.

[matttbe]

Indeed, reducing CPU overhead is always a win!

1. `MSG_ZEROCOPY` on the send path (TCP/UDP, `setsockopt(SO_ZEROCOPY)` + `send()` with `MSG_ZEROCOPY` flag) — lowest effort, kernel 4.14+

I guess that will be optional, right? :) (MPTCP doesn't support it yet, and not sure whether it really needs to in fact)

sendfile() with a pre-filled temp file — avoids userspace buffer entirely, but means we'd need to manage a scratch file

If I remember well, the file in sendfile is about file descriptors, it doesn't have to be a file on a disk. I think iperf3 is using it with mmap()

Any sense of what throughput range you're testing on the embedded side? Curious whether the copy overhead is showing up as CPU saturation or as a throughput ceiling.

I remember that a while ago, I was using iperf3 on a now old router with a MIPS processor, and the difference was quite visible. Something like 30Mbps vs 100Mbps.