Buffer space error with example client

[plauche]

Hi,

I tried out the example UDT client code and I keep getting this error Error: Os { code: 55, kind: Uncategorized, message: "No buffer space available" }. I'm running macOS 13.1 on a fairly modern machine, and have run into this error before and after rebooting, so I don't think it's an issue with local resource availability.

I did some digging and traced the error down to tokio-udt/src/multiplexer.rs:41. Have you run into this error when using this library?

Example code used:

use std::net::Ipv4Addr;
use tokio::io::{AsyncWriteExt, Result};
use tokio_udt::UdtConnection;

#[tokio::main]
async fn main() -> Result<()> {
    let port = 9000;
    let mut connection = UdtConnection::connect((Ipv4Addr::LOCALHOST, port), None).await?;
    loop {
        connection.write_all(b"Hello World!").await?;
    }
}

[CryptoGladi]

I checked this code on my computer and it works

Info about my PC

OS: openSUSE (Linux)
Kernel: 6.1.6-1-default

This is most likely a bug of MacOS.

In addition, this repository seems to be already forgotten. I submitted a pull request but it hasn't been accepted yet.

[fivebats]

@plauche This appears to be your Mac OS host not liking the default configuration parameters.

If you explicitly configure the connection like this:

use std::net::Ipv4Addr;
use tokio::io::{AsyncWriteExt, Result};
use tokio_udt::{UdtConfiguration, UdtConnection};

#[tokio::main]
async fn main() -> Result<()> {
    let port = 9000;
    let mut conf = UdtConfiguration::default();
    let sz = 6_000_000;
    conf.udp_snd_buf_size = sz;
    conf.udp_rcv_buf_size = sz;
    let mut connection = UdtConnection::connect((Ipv4Addr::LOCALHOST, port), Some(conf)).await?;
    loop {
        connection.write_all(b"Hello World!").await?;
    }
}

In sender and receiver it may work OK. That was just a quick test I made to make it run on a host that has not been tuned for udp. There may be other changes in the configuration needed depending on what you are doing, and of course you can change a lot of the kernel settings using sysctl.

So I think the real problem is that the default connection configuration doesn't work on the default settings of Mac OS.

[CryptoGladi]

@fivebats, so the error is due to a too big buffer? I found a similar error.

It turns out you just need to change their value to smaller ones?

tokio-udt/src/configuration.rs

Line 5 in 450cfcc

const DEFAULT_UDP_BUF_SIZE: usize = 8_000_000;

I don't have a macbook so I can't test this theory. I can install macOS on virtual work tomorrow.
P.S. I think you need to create a fork of this library for this. It is forgotten.

[fivebats]

@CryptoGladi Yes, changing DEFAULT_UDP_BUF_SIZE to a lower value, such as 6_000_000 should accomplish the same thing as my code above, at the expense of the users on linux. Perhaps the defaults can be conditionally compiled for each platform.

I'm not sure I would call any of this an error. Linux and BSD-derived kernels just diverge a bit in how they handle network buffering. Here is an old, but I think still valid, discussion of how to increase the udp buffer size on Mac OS. There are a few interacting settings.

Note that I'm deploying on Linux, but developing on a Mac so it is easy for me to test things on a Mac. I probably won't discover any Mac problems with this crate since my code mostly is running on Linux.

[CryptoGladi]

@plauche Why don't you close the issues?

You can solve your problem like this:

mod from_udt_tokio {
    pub(crate) const DEFAULT_MSS: u32 = 1500;
    pub(crate) const DEFAULT_UDT_BUF_SIZE: u32 = 81920;
    pub(crate) const DEFAULT_UDP_BUF_SIZE: usize = 81920; // CHANGED!
}

/// Config for [`tokio-udt`](https://github.com/Distributed-EPFL/tokio-udt)
///
/// Solves a [bug](https://github.com/Distributed-EPFL/tokio-udt/issues/2) in MacOS
pub(crate) const UDT_CONFIGURATION: Option<UdtConfiguration> = Some(UdtConfiguration {
    mss: from_udt_tokio::DEFAULT_MSS,
    flight_flag_size: 256_000,
    snd_buf_size: from_udt_tokio::DEFAULT_UDT_BUF_SIZE,
    rcv_buf_size: from_udt_tokio::DEFAULT_UDT_BUF_SIZE * 2,
    udp_snd_buf_size: from_udt_tokio::DEFAULT_UDP_BUF_SIZE,
    udp_rcv_buf_size: from_udt_tokio::DEFAULT_UDP_BUF_SIZE,
    udp_reuse_port: false,
    linger_timeout: Some(Duration::from_secs(10)),
    reuse_mux: true,
    rendezvous: false,
    accept_queue_size: 1000,
});

Do you want to fix this bug in this repository?

P.S. When I have time, I will rewrite this library. It will not be soon!