QoS Questions

[mechaniputer]

I am writing a simple app which involves two machines using FastDDS to exchange samples. The first machine runs a "driver" and a "listener" process, while the other machine runs a single "server" process.

My code is derived from the HelloWorld example. Here is my code: https://github.com/mechaniputer/LatencyBench-FastDDS

Here is what these processes are supposed to do. The driver should generate samples at regular time intervals and send them on a topic that the server is subscribed to. When the server receives a sample, it should forward the sample by writing it to a second topic which the listener is subscribed to.

In short, it's a round trip test between 2 machines, with one of the machines intentionally bottlenecked by using a single process while the other machine splits sending and receiving between two processes via separate topics.

Here is the problem I am having. Even though I am using a reliable QoS, the listener only receives about half of the samples that the driver sent. Is the QoS set up wrong?

Edit: An update. I changed the writers in both the "driver" and the "server" app to use KEEP_ALL_HISTORY_QOS. It runs for a few seconds and then the driver slows down drastically while the server seems to stop receiving any samples and times out.

[mechaniputer]

I got it to work like I intended, but it is much slower. I applied the same durability and history settings to all readers and writers.
KEEP_ALL_HISTORY_QOS, TRANSIENT_LOCAL_DURABILITY_QOS.

No samples get lost now. I'm wondering if there is a more performant way of achieving this though. Advice appreciated.

[mechaniputer]

I have made further changes and am once again losing roughly 10% of samples from the driver to the server app. My main change was to send an initial sample indicating how many samples the driver is going to send (for accurate CPU% measurement over the test interval).

Edit:

It appears that the samples are being lost early in the test. Perhaps the server is missing some samples while it prepares to enter the main loop? But the QoS settings I'm using should prevent that, right?

[mechaniputer]

I thought maybe the issue was that I was not checking for DataWriter->write() returning errors. So I added a loop to retry the write until it returns RETCODE_OK. This had unintended effects. It looks like the server app is repeatedly sending the same sample to the listener app while write() returns an error. That is, the server indefinitely retries the same sample, which is actually being received repeatedly despite the write() return value indicating failure.

[mechaniputer]

I think I've solved it. The driver was terminating after it had sent everything, but it needs to keep running to handle retries? I assume retries are being done using interrupts?

[MiguelCompany]

There is this API you can use on the writer

[MiguelCompany]

Retries are done on a separate thread

[mechaniputer]

Thank you, this is exactly what I needed! Seems to work fine now.

One last thing. I am wondering if there is a way to improve the performance while maintaining these delivery guarantees. I am getting about 3000 samples per second throughput with 64B payload size. Can it do better?

[mechaniputer]

I managed to get 40,000 samples per second by throttling it. If I try to go for 50,000/sec then it drops to about 3000 again. I guess it gets bogged down with retries if it tries to run too fast.

[mechaniputer]

Summary: The lost samples were in fact incomplete retries and I needed to keep the app running. The DataWriter's wait_for_acknowledgments() provides a nice way to do this. Thanks.