noop http benchmark - what am I doing wrong? #1311
-
BackgroundI've created a simple network server that
So it is just a noop http benchmark to find how much I can get Liburing usage
ClientsI've mostly used wrk2 as load gen tool, but have also occasionally used Machine detailsuname -a
Linux ryzen5700x 6.8.0-51-generic #52-Ubuntu SMP PREEMPT_DYNAMIC Thu Dec 5 13:09:44 UTC 2024 x86_64 x86_64 x86_64 GNU/LinuxModel name is cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.1 LTS"Benchmark and ResultsI am using wrk2 to simulate 20480 simultaneous connections and pump a load of 102400 RPS on this server via localhost. ./wrk -c 20480 -t 6 -d 300s http://localhost:7777 -R 102400 -L
Running 5m test @ http://localhost:7777
6 threads and 20480 connections
...(removed for verbosity)
Thread Stats Avg Stdev Max +/- Stdev
Latency 651.79ms 4.15s 49.97s 97.94%
Req/Sec 17.09k 391.18 18.54k 69.22%
Latency Distribution (HdrHistogram - Recorded Latency)
50.000% 106.24ms
75.000% 109.18ms
90.000% 112.32ms
99.000% 26.64s
99.900% 46.73s
99.990% 49.09s
99.999% 49.74s
100.000% 50.00s
(...removed detailed percentile spectrum)
#[Mean = 651.787, StdDeviation = 4150.317]
#[Max = 49971.200, Total count = 27913044]
#[Buckets = 27, SubBuckets = 2048]
----------------------------------------------------------
29784131 requests in 5.00m, 2.58GB read
Socket errors: connect 0, read 0, write 0, timeout 6045
Requests/sec: 99276.24
Transfer/sec: 8.80MBObservations
I am interested mostly in single core performance. I've also tried Questions
More Questions(unrelated - feel free to ignore)
|
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 16 replies
-
|
That's quite a good report with all details!
25s and 50s at for 99% and 99.9% sounds insanely high. I'm highly suspicious of that.
It's pretty easy to saturate a CPU, so I wouldn't worry too much about it unless performance is also low.
The generic (non io_uring) tx path here is pretty expensive, around 60% CPU. The good news is it's because of loopback, so it includes cycles spent processing the rx path of the other end (~30%). You also got maybe ~10% of cycles consumed by apparmor, and usually people don't care about it much and it's there just because of the stock kernel and distro defaults. And then you'll be left with the fact that the net stack is heavy. If each request drives only a handful of bytes you won't get anywhere close to the NIC speeds. It'd need multiplexing instead of one send+recv pair per connection, on top of which you can additionally put multishot requests and so on. I also see some accounting overhead, if you regenerate the profile for a new kernel (~6.13) I can take another look. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks @isilence . I don't have any experience in the linux kernel, so please bear my ignorance.
Oh...I had no idea about that - this was very helpful. 1 question - this behaviour(server processing the rx path of the other end of loopback client) is irrespective of io_uring? i.e it always happens regardless of whether application uses io_uring or not? So for a real client on the network, this overhead shouldn't be there? Right now I don't have the setup to put the client on the network(too ashamed to admit that), will try it as soon I can.
Yeah, I guess I don't want to touch apparmor
There is multiplexing of I/O from/across multiple connections already, but how is it possible to multiplex I/O for a single connection(especially for http/1.1 - I guess for http/2 multi-shot recv would be very helpful). I mean, first we need to do the // serving a connection
while (true) {
recv(...)
// parse and validate
....
send(...) // response
}I did think to batch the submit for multi-shot
I was running ebpf based profiler to collect cpu profile. I am curious, how did you catch that(accounting overhead?) from the cpu profile?
Can't do that right now, haven't compiled kernel(ever since I left college) - don't want to risk breaking my system right now.
Yeah, I completel agree. std::fill(reg_files_, reg_files_ + max_open_files, -1);
rc = io_uring_register_files(&ring_, reg_files_,
static_cast<uint32_t>(max_open_files)); // API fails if max_open_files as more than (1 << 20)
Thank you so much for your help and patience. |
Beta Was this translation helpful? Give feedback.
-
Right, it'll be there irrelevant of io_uring. With a real network card you could say that overhead will be moved to rx path of the server where the other end is.
There will be the driver overhead instead, but it's normally lower.
I'd say benchmarks are only useful when they can approximate well enough parts of the real world that we care about. Do you really care about this workload and/or http/1.1? http/1.1 -> 2.0 transition should normally be more impactful than optimisations in IO engine, and maybe then the benchmark doesn't measure the properties you want. Regardless I'd need to refresh myself on what http feature came in what version.
Right, but e.g. when you load a page the client requests lots of resources at once (spiky traffic), and IIRC it all multiplexed in http/2 and served via a single socket. And if so a single multishot receive for several app level requests, then the user can process all that and response to all the requests with a single sendmsg assuming the protocol allows it. Really depends what your target workload is.
Links can be problematic, but you shouldn't need them here at all unless I missed sth. You can even queue a new receive request as soon as you find the previous one has died even before sending a response.
I should've been more specific, the accounting I mentioned is network cgroup accounting, which is expected to be there, but maybe that can be optimised if it's not already done in more recent kernels.
It should be of low risk if you install 6.12 (surely you don't want rc kernels) as an additional kernel without removing the current one, so if something doesn't work you just reboot into the old one. (you can default grub to the old one for convenience). Though I'm not insisting, everything has risks, even the installation stage can be accidentally fat fingered.
I know nothing about it
You can just use normal fds instead. For a long time registered file had a nasty pitfall because of which it was discouraged from using them with long lived IO, i.e. requests using a registered file that is not expected to complete in short-ish bound amount of time. It has been fixed only in quite recently, need to look it up but that could be 6.12 if not in not yet released 6.13. And there has always been some cost on registering and removing a file from io_uring. So if it's a really cold connection you probably don't need registered files, but it might make sense if a connection is slow but still drives a good amount of bytes during lifetime and/or bursty. In short, you can measure if it really improves performance in a real app...
... but regardless it might make sense to lift / increase the limit, there is nothing fundamental about it. |
Beta Was this translation helpful? Give feedback.
-
|
This was extremely helpful. Thank you Pavel I agree that micro-benchmarks are not particularly useful. My objective with this was to get enough experience on how to structure the application around liburing, since completion based model is different than epoll(readyness based) and there are quite a lot of APIs and performance tradeoffs(e.g batching at both submit and completion side). Is there a place(other than linux source code) where I get get to know about the how io_uring is implemented inside the linux kernel. |
Beta Was this translation helpful? Give feedback.
-
That's fair and useful. It can serve as a baseline, but I just wouldn't make any final conclusions if the protocol indeed is a limiting factor for your use case. One thing though that is very interesting in your numbers is the insanely high 99+% latency. io_uring tries not to be unfair, there shouldn't be cases where a packet is semi infinitely stuck waiting while e.g. the kernel processing newer data. It's definitely worth looking into, e.g. you can measure how long CQE processing takes, and we can trace more info from the kernel as well. Let know if you find anything.
I think this article should be the most relevant for net app writers https://github.com/axboe/liburing/wiki/io_uring-and-networking-in-2023 |
Beta Was this translation helpful? Give feedback.
-
I suspect it is because of the some problem with the client - that's why I asked this question.
I tried 2 other clients and they don't give such huge numbers and so I am going to ignore those for now I started implementing multi-shot + io_uring_sqe* sqe = io_uring_get_sqe(&io_service_->ring_);
+ if (sqe == nullptr) {
+ // log error
+ abort();
+ }
+ io_uring_prep_multishot_accept_direct(sqe, fd_, peer_addr_, addr_len_,
+ flags_);
+ io_uring_sqe_set_data(sqe, this);
Rest of the stuff is more or less the same.
What did I do wrong, rest of the code is same - I quickly tested the As noted in my original post, I am using direct descriptors, can that be the reason? |
Beta Was this translation helpful? Give feedback.
-
That's my first reaction as well.
So, you're saying it only happens with (multishot) accepts and you specifically tested with vs without? |
Beta Was this translation helpful? Give feedback.
-
Both are arbitrary limits the kernel enforces. Do you really need that much? If you get 64K completions in the ring then your app is doing something very wrong, the latency of processing them will be insanely high. And a ring (single CPU) won't be able to handle 1M hot connections, so most of them are probably cold and you don't really need fixed files at all unless you get some wins from |
Beta Was this translation helpful? Give feedback.
Right, it'll be there irrelevant of io_uring. With a real network card you could say that overhead will be moved to rx path of the server where the other end is.