Slow UDP over 100Gb NIC

[VolunPotest]

Hi!
We have in lab 100Gb NIC, and we are trying to measure TCP and UDP throughput.

The max we can get for TCP ~ 50 GBits/s
The max we can get with UDP ~ 12 Gbit/s

My question is:
Why is UDP throughput so low (comparing to TCP). I heard a lot that UDP should be much faster than TCP, but in real example we have the opposite.
If there is any explanation about that? Is it connected with Linux Kernel itself or there is a problem with iperf3?

We usually use basic iperf3 commands like:
iperf3 -c [server_ip] -t 30 -V -u -b 0

• Version of iperf3: 3.7
• Operating system (and distribution, if any): Ubuntu 20.04 LTS

Thank you!

[davidBar-On]

This may be because of the packets sizes - the default for TCP and UPD are different. Use the -d option to see the value used - the len value in get_parameters JSON.

To compare apples to apples, the same packet size should be used for both, using the -l option.

[VolunPotest]

Hi!
Thank you for your answer.

Packet size for TCP is 128K (by default) and for UDP is 1448 bytes. (MTU 1500), but as far as I understand, NIC will split this big TCP packet for small ~1500 bytes chunks (because packet size couldn't be more than MTU size).

Okay. I've tried to increase packet size for UDP with -l option and maximum is 64K, but I can see that is not UDP packets anymore, it's IP fragments packets, but throughput is ~15 Gbits/s now. (Not huge difference)

I also tried to make an opposite. I've made the packet size for TCP and UDP equal to 1400:

• For TCP results is ~6Gbits./s
• For UDP results is ~4Gbit/s.

Still, UDP is lower than TCP

[lomaxfrog]

Hi, If you are using Mellanox ConnectX5, it might be that you are running into the issue that only the first ethernet frame of your packet get properly dispatched into the receiving queues of the NIC (load balancing the frames on different IRQ/cores) and that the remaining frames all go to queue 0, i.e. to the same IRQ/core. You can verify that if you see a single core on the receive side being max'ed up, or looking at interrupt counts and see if a queue is overwhelmed. Eric

…

On Fri, Jan 28, 2022 at 9:03 AM 4ipers ***@***.***> wrote: Hi! Thank you for your answer. Packet size for TCP is 128K (by default) and for UDP is 1448 bytes. (MTU 1500), but as far as I understand, NIC will split this big TCP packet for small ~1500 bytes chunks (because packet size couldn't be more than MTU size). Okay. I've tried to increase packet size for UDP with -l option and maximum is 64K, but I can see that is not UDP packets anymore, it's IP fragments packets. I also tried to make an opposite. I've made the packet size for TCP and UDP equal to 1400: - For TCP results is ~6Gbits./s - For UDP results is ~4Gbit/s. Still, UDP is lower than TCP — Reply to this email directly, view it on GitHub <#1271 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABSGNCMKRJJYQ75PFILWJ4LUYLD75ANCNFSM5M4EPSZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[VolunPotest]

Thank you for the answer, @lomaxfrog.
I don't remember the model name, but I think it's intel network card.

I didn't really understand what you're trying to tell me about IRQ/core.
Could you please recommend me an article/guidance or something like that to understand this issue more properly?

Thank you

[swlars]

Hi,

A bigger socket buffer size might help as well, using the -w option.

[kgersen]

check this: https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size