Throughput example receive many out of order samples

[allenwang-git]

When I clone the cyclonedds and cyclonedds-cxx from their current master branches, and run cyclonedds-cxx throughput example with the commands cyclonedds-cxx/build/bin/cxxThroughputPublisher 8192 0 1 0 "Throughput example" , cyclonedds-cxx/build/bin/cxxThroughputSubscriber 0 0 "Throughput example", I got such out put form subscriber side:

=== [Subscriber] Cycles: 0 | PollingDelay: 0 ms | Partition: Throughput example
=== [Subscriber] Using a waitset approach.

=== [Subscriber] Waiting for a writer ...
=== [Subscriber] Waiting for samples...
=== [Subscriber] Received 608 samples totalling 637534208 bytes in 1 seconds | Rates: 607.7 samples/s, 5098 Mbit/s, with 596 samples received out of order.
=== [Subscriber] Received 611 samples totalling 640679936 bytes in 1.001 seconds | Rates: 610.2 samples/s, 5119 Mbit/s, with 1194 samples received out of order.
=== [Subscriber] Received 609 samples totalling 638582784 bytes in 1.001 seconds | Rates: 608.4 samples/s, 5104 Mbit/s, with 1791 samples received out of order.
=== [Subscriber] Received 608 samples totalling 637534208 bytes in 1.001 seconds | Rates: 607.4 samples/s, 5095 Mbit/s, with 2390 samples received out of order.
=== [Subscriber] Received 609 samples totalling 638582784 bytes in 1 seconds | Rates: 608.7 samples/s, 5107 Mbit/s, with 2993 samples received out of order.
=== [Subscriber] Received 613 samples totalling 642777088 bytes in 1.002 seconds | Rates: 611.7 samples/s, 5131 Mbit/s, with 3595 samples received out of order.
=== [Subscriber] Received 607 samples totalling 636485632 bytes in 1.002 seconds | Rates: 605.8 samples/s, 5082 Mbit/s, with 4195 samples received out of order.
=== [Subscriber] Received 620 samples totalling 650117120 bytes in 1.001 seconds | Rates: 619.4 samples/s, 5196 Mbit/s, with 4802 samples received out of order.

As u can see, many out of order samples are received. I didnt modify any line of source code or this example, could anyone tell me the reason why the output looks not good. when i run the Throughput example in CycloneDDS repo, the output is normal, without any out of order samples. @Pro @k0ekk0ek @ichernev @chrisguindon @kydos

[allenwang-git]

That's weird because these commands are from your example README. However, if I directly use cyclonedds-cxx/build/bin/cxxThroughputPublisher , cyclonedds-cxx/build/bin/cxxThroughputSubscriber, then the output will be:

=== [Subscriber] Cycles: 0 | PollingDelay: -1 ms | Partition: Throughput example
=== [Subscriber] Using a listener approach.

=== [Subscriber] Waiting for a writer ...
=== [Subscriber] Waiting for samples...
=== [Subscriber] Received 113 samples totalling 933888 bytes in 1.001 seconds | Rates: 115.9 samples/s, 7.597 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 99 samples totalling 811008 bytes in 1.001 seconds | Rates: 99.93 samples/s, 6.549 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 100 samples totalling 827392 bytes in 1.001 seconds | Rates: 100.9 samples/s, 6.614 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 100 samples totalling 819200 bytes in 1.001 seconds | Rates: 99.93 samples/s, 6.549 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 99 samples totalling 811008 bytes in 1.001 seconds | Rates: 99.93 samples/s, 6.549 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 100 samples totalling 819200 bytes in 1.001 seconds | Rates: 99.93 samples/s, 6.549 Mbit/s, with 0 samples received out of order.
=== [Subscriber] Received 100 samples totalling 819200 bytes in 1.001 seconds | Rates: 99.93 samples/s, 6.549 Mbit/s, with 0 samples received out of order.

which did not have out of order samples but the speed is super slow compared to the above one, do u know the reason?

[eboasson]

The behaviour exactly matches what you get when the reliability QoS is set to "best effort" and/or the history QoS is set to "keep last" instead of "keep last". When I build with current master, cyclonedds ls -q shows both the reliability and the history QoS being set like that.

So that explains ... I do recall a bug with the QoS settings got introduced in the C++ example some time ago. I somehow believed it got fixed, but apparently I imagined that. Oops.

When you use a listener, it processes the incoming data on the same thread as the one receiving it from the network, instead of on a separate thread. That effectively eliminates the "keep last" problem (at least the problem of having "keep last" on the reader side). So the difference between the behaviour in the two cases is ok.

Thanks for bringing it to our attention.

Finally, please don't ping random people: none of those you mentioned are relevant for this issue and those who do are keeping an eye on the issues that get created.

[eboasson]

Ah, found it: #404 ... reviewed, not merged yet because of a detail unrelated to the QoS settings ...

[allenwang-git]

Thanks @eboasson and Sorry for calling all contributors. I didnt dive deep in qos settings. My question, in a word, is there any way we can keep this example working with high transfer rate without any out of order messages? Even if 5000-6000Mbit/s is still lower than what I observed in Cyconeddes/Throughput example, which can reach 8000-9000Mbit/s. That's not a good news to a cpp user. Thanks in advance

[eboasson]

I just merged fixed the remaining problems with #404 and merged it. So the problem should disappear when you update.

There is also a PR open for improving the performance of the serializer in C++ (#361). It looks like I may have overlooked some updates there and left it hanging while in fact it is ready for merging. I'll have another look, but do feel free to try it out.

[allenwang-git]

Cool, I have another question after testing the new master branch, with and without shm support. I'm surprised that the transfer rate is almost the same no matter we use iceoryx to support shm or not. However, this rate could be improved to 10 times by using iceoryx shm support in CycloneDDS repo.
Below is the Cyclonedds output results, first is the result running shmThroughput and the second is running Throughput, both in cyclonedds repo

I also attached the out put of Cyclonedds-cxx repo Throughput example results below, first used shm, second didnt:

Do u have any idea? @eboasson

[allenwang-git]

nvm, I realize this Throughput example cannot run with shm, I'm trying to write a ShmThroughput example for cycloneDDS. lmk if I can make a pr to push the work