fix: solve Offset Manager and Worker racing condition when stopping

[filipeesch]

Description

There was a racing condition between the Offset Manager and the Consumer Worker, which needed to be fixed. The Cancellation token that was passed to the EnqueueAsync worker method was removed, which meant that the worker would accept the message before stopping. Previously, the behaviour was to cancel the last message delivery, but to do this, we needed to try to enqueue the message in the worker and then enqueue the message in the Offset Manager. Calling the method in this order caused a racing condition, as the worker could process the message before it was enqueued in the Offset Manager, triggering an exception and stopping the consumer.

Fixes #482 #488

How Has This Been Tested?

The tests were manually executed and the automated tests were run.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have added tests to cover my changes
• I have made corresponding changes to the documentation

Disclaimer

By sending us your contributions, you are agreeing that your contribution is made subject to the terms of our Contributor Ownership Statement