Raise NotNotReadyError from KafkaConsumer.poll after a retry timeout if the KafkaBroker is not available

[zembunia]

KafkaConsumer.poll never returns in case of no connection with the KafkaBroker. This exception gives a chance to the outer program to know about the problem.

[zembunia]

Please only review the comparison of the files because the intermediate 48 commits are for getting to know what to change.

[dpkp]

you know there's always git rebase -i master -- squash those commits :)

[dpkp]

I appreciate the PR, but I'm not a fan of this change. We try to stay fairly consistent with the java client configuration options so I dont like to add new options that dont exist upstream unless absolutely necessary.

It looks like the problem you are trying to solve is that ensure_coordinator_known is blocking and does not support a timeout. Have you checked how the java client handles this? Perhaps a simpler solution would be to add an optional timeout to the function itself and check it during the while loop. Is there a particular upstream function call that you are having problems with vis-a-vis blocking?

[zembunia]

Java client has the same behavior reported as an issue. (https://issues.apache.org/jira/browse/KAFKA-1894) I will also test it with the same scenario.

To get the messages in a topic I use KafkaConsumer.poll and it never returns if the broker is not there. How can I understand it is not there?
I have seen that iterator interface also calls ``ensure_coordinator_known``` which will also block.

[zembunia]

Java consumer is has a "consumer.timeout.ms" configuration option which lets ConsumerIterator.hasNext() to timeout and raise ConsumerTimeoutException. Its default value is -1 so that it never times out.

[jeffwidman]

@zembunia if you're still interested in this PR, can you rebase and cleanup the commit history (see comment above about squashing) so that it's easier to review/discuss?

[zembunia]

#1209 replaces this PR