ESPNow - Disabling ACK packets increases latency?

[charlielee206]

Hi, I am trying to send audio over ESPNOW, and for more throughput and less latency, I have decided to try disabling acknowledgement packets. My theory was that by skipping the whole handshake phase, we could achieve more throughput and less latency for the cost of less audio quality or artifacts.
However, when I actually implemented the changes, something strange happened. Although I have achieved more throughput, the latency increased dramatically.

I also noticed that the latency increased over time. Over a period of about 20 seconds right after power cycling both the transmitter and the receiver at the same time, I noticed that there was a gradual increase in delay.

For my tests, the default latency (when ACK packets were enabled) was about 25ms. When disabled the ACK packets, the latency jumped to about 60ms on startup, which gradually increased to about 80ms.

I have disabled the ACK packets by doing cfg.use_send_ack = false; for the config of the ESPNowStream in the main sketch. I measured the throughput by communication-espnow-send and receive-measure scripts, and I measured the latency by actually sending audio over espnow, and comparing the time between the source and the outputs.

Also, while trying to solve this issue, I have made a small change in default_recv_cb, within AudioTools/Communication/ESPNowStream.h. I removed the delay while waiting for the buffer. I think this just drops the packets when there is no space in the buffer, but doing this reduced the latency buildup without noticeable audio defects. (I guess most of the dropped packets were defective?)

within `default_recv_cb` in `ESPNowStream.h`:

changed from

while (bufferAvailableForWrite() < data_len) {
      delay(2);
    }

to

    if (ESPNowStreamSelf->cfg.use_send_ack) {
      while (bufferAvailableForWrite() < data_len) {
        delay(2);
      }
    } else {
      if (bufferAvailableForWrite() < data_len) {
        return;
      }
    }

Although this did remove most of the latency buildup, It did not remove the original increase in latency. Now, at least in my setup, the latency stays at around 60ms.

I think the latency comes from a blocking function somewhere that waits for the now non-existant ACK packets. However, although I might be missing something, I could not find any blocking functions that expect ACK packets within the audio-tools library. I also could not look through the main ESPNOW code within the core library, since they exist in a pre-compiled archive file. (All i could find that was directly read and editable were just header files, which all point to functions within the precompiled archive file. libespnow.a to be exact)

How would I remove the extra latency that happens when use_send_ack is set to true? I have tried everything I could think of, and I have no idea where I should look next. Some tips would be appreciated.

[pschatzmann]

If the latency is too big, you should try to minimize the buffers in the whole processing chain: e.g. the buffer_size and buffer_count in I2S and ESPNow are a bit big to be on the save side and you can try to optimize this considerably.

The use_send_ack in ESPNow is critical to prevent a buffer overflow assuming the sending speed is above the playback speed. If you remove it you need to do 2 things:

• Make sure that you limit the sending speed to the sample rate
• Add some additional logic on the output side to prevent buffer over and underruns because of the slight clock differences between the sender and receiver: e.g. Monitor the buffer and do some resampling to speed up or slow down the playback slightly based on the filling grade of this buffer.

Alternatively you could try to save the received data in PSRAM if the sending file is small enought. Maybe it is also an option to save it in a File, but I don't know about the limitations of writing and reading from a file at the same time.

ps. I think the delay(1) is important to process some other tasks.
You can try to inclrease the thruput by selecting a higher wifi_phy_rate_t

[charlielee206]

I have minimized the buffers. There is probably some more that I could shave off, but what I'm questioning is why disabling ACK would increase latency.
Why would buffer overflow create latency, even when I made it so that the code would drop extra packets? I would understand if I heard static, cuts, or general audio defects, but I don't understand why latency would increase.
Since the sample frequency and channels stayed the same across the experiments, and I'm dropping additional packets, I don't think it would be a problem where the CPU gets overwhelmed with too much data.
I'm still trying to wrap my head around this problem, since I still think that I should have less latency with lower quality.
Is there something obvious I'm missing?

[pschatzmann]

My guess is, that thisis rather caused by your removal of the delay as mentioned above.

[charlielee206]

If I keep the delay in the code, the Initial latency stays the same as when I removed the delay in the code, but the latency ramps up over time. My thought was that the initial spike in latency is caused by something else, and the additional latency that ramps up is due to the buffer waiting longer and longer as more packets stack in the buffer.
Removing the delay in the code solved the increasing latency problem, but not the initial spike in latency.
My current thought is that the initial spike is caused by something else, but the additional latency rise was caused by buffers. Removing the delay will remove latency problems related to the buffer, at the cost of audio quality.
And my test results seem to back this hypothesis.

Results

Setting	Latency-on boot	Latency-after 60seconds
Stock	25ms	25ms
No ACK	60ms	90ms
No ACK, Modified Library (delay removal)	60ms	60ms

I'm sure that removing the delay will cause many problems in the long run, and generally compromise the stability of the code. But I'm not sure why it would cause extra latency problems. And if removing the delay will create more latency, why did the latency after 60 seconds drop?

[pschatzmann]

I guess that is the time it takes to fill up the buffer and then instead of receiving any data most of the effort is spent on waiting

[pschatzmann]

I changed the logic to use a semaphore to manage the confirmations and the BufferRTOS class for the buffer.