Skip to content

Releases: aio-libs/aiokafka

Speed enhancements and Consumer rewrite

01 Feb 13:02
Compare
Choose a tag to compare

Major changes:

  • Full refactor of the internals of AIOKafkaConsumer. Needed to avoid several
    race conditions in code (PR #286, fixes #258, #264 and #261)
  • Rewrote Records parsing protocol to allow implementation of newer protocol
    versions later
  • Added C extension for Records parsing protocol, boosting the speed of
    produce/consume routines significantly
  • Added an experimental batch producer API for unique cases, where user want's
    to control batching himself (by @shargan)

Minor changes:

  • Add timestamp field to produced message's metadata. This is needed to find
    LOG_APPEND_TIME configured timestamps.
  • Consumer.seek() and similar API's now raise proper ValueError's on
    validation failure instead of AssertionError.

Bug fixes:

  • Fix connections_max_idle_ms option, as earlier it was only applied to
    bootstrap socket. (PR #299)
  • Fix consumer.stop() side effect of logging an exception
    ConsumerStoppedError (issue #263)
  • Problem with Producer not able to recover from broker failure (issue #267)
  • Traceback containing duplicate entries due to exception sharing (PR #247
    by @Artimi)
  • Concurrent record consumption rasing InvalidStateError('Exception is not set.') (PR #249 by @aerkert)
  • Don't fail GroupCoordinator._on_join_prepare() if commit_offset()
    throws exception (PR #230 by @shargan)
  • Send session_timeout_ms to GroupCoordinator constructor (PR #229 by @shargan)

Big thanks to:

  • @shargan for Producer speed enhancements and the batch produce API
    proposal/implementation.
  • @vineet-rh and other contributors for constant feedback on Consumer
    problems, leading to the refactor mentioned above.

Minor bugfix release

01 Feb 13:00
Compare
Choose a tag to compare
  • Added AIOKafkaProducer.flush() method. (PR #209 by @vineet-rh)
  • Fixed a bug with uvloop involving float("inf") for timeout. (PR #210 by
    dmitry-moroz)
  • Changed test runner to allow running tests on OSX. (PR #213 by @shargan)

Major docs upgrade. Many consumer fixes including new API's.

17 Aug 11:59
Compare
Choose a tag to compare

Starting from 0.3.0 we discourage imports from kafka namespace. Starting from 0.4.0 aiokafka will stop supporting objects imported from kafka namespace. Please import from aiokafka namespaces instead.

  • Moved all public structures and errors to aiokafka namespace. You will no
    longer need to import from kafka namespace.
  • Changed ConsumerRebalanceListener to support either function or coroutine
    for on_partitions_assigned and on_partitions_revoked callbacks. (PR #190
    by @ask)
  • Added support for offsets_for_times, beginning_offsets, end_offsets
    API's. (issue #164)
  • Coordinator requests are now sent using a separate socket. Fixes slow commit
    issue. (issuer #137, issue #128)
  • Added seek_to_end, seek_to_beginning API's. (issue #154)
  • Updated documentation to provide more useful usage guide on both Consumer and
    Producer interface.

Fix for Producer to handle retries correctly on node failures

23 Jul 10:42
Compare
Choose a tag to compare
  • Fixed retry problem in Producer, when buffer is not reset to 0 offset.
    Thanks to @ngavrysh for the fix in Tubular/aiokafka fork. (issue #184)
  • Fixed how Producer handles retries on Leader node failure. It just did not
    work before... Thanks to @blugowski for the help in locating the problem.
    (issue #176, issue #173)
  • Fixed degrade in v0.2.2 on Consumer with no group_id. (issue #166)

More bugfix for Consumer and reconnects

17 Apr 12:07
Compare
Choose a tag to compare
  • Force reconnect after KafkaTimeoutException. (PR #149 by @Artimi)
  • Fixed compacted topic handling. It could skip messages if some of them were
    compacted (issue #71)
  • Fixed old issue with new topics not visible in pattern subscription (issue #46)
  • Another fix for Consumer race condition on JoinGroup. This forces Leader to
    wait for new metadata before assigning partitions. (issue #118)
  • Changed metadata listener in Coordinator to avoid 2 rejoins in a rare
    condition (issue #108)
  • getmany will no longer return 0 results before timeout. (issue #117)

Big thanks to @Artimi for pointing out several of those issues.

Consumer enhancements and coordinator bugfix.

19 Feb 13:03
Compare
Choose a tag to compare
  • Add a check to wait topic autocreation in Consumer, instead of raising UnknownTopicOrPartitionError (PR #92 by fabregas)
  • Consumer now stops consumption after consumer.stop() call. Any new get* calls will result in ConsumerStoppedError (PR #81)
  • Added exclude_internal_topics option for Consumer (PR #111)
  • Better support for pattern subscription when used with group_id (part of PR #111)
  • Fix for Consumer subscribe and JoinGroup race condition (issue #88). Coordinator will now notice subscription changes during rebalance and will join group again. (PR #106)
  • Changed logging messages according to KAFKA-3318. Now INFO level should be less messy and more informative. (PR #110)
  • Add support for connections_max_idle_ms config (PR #113)

SSL support and bugfixes

18 Dec 22:12
Compare
Choose a tag to compare
  • Added SSL support. (PR #81 by Drizzt1991)
  • Fixed UnknownTopicOrPartitionError error on first message for autocreated topic (PR #96 by fabregas)
  • Fixed next_record recursion (PR #94 by fabregas)
  • Fixed Heartbeat fail if no consumers (PR #92 by fabregas)
  • Added docs addressing kafka-python and aiokafka differences (PR #70 by Drizzt1991)
  • Added max_poll_records option for Consumer (PR #72 by Drizzt1991)
  • Fix kafka-python typos in docs (PR #69 by jeffwidman)
  • Topics and partitions are now randomized on each Fetch request (PR #66 by Drizzt1991)

Updated Kafka to 0.10.0.1 and kafka-python to 1.3.1

07 Nov 13:10
Compare
Choose a tag to compare
  • Bumped python-kafka version to 1.3.1 and Kafka to 0.10.1.0.
  • Fixed auto version detection, to correctly handle 0.10.0.0 version
  • Updated Fetch and Produce requests to use v2 with v0.10.0 message format on brokers.
    This allows a timestamp to be associated with messages.
  • Changed lz4 compression framing, as it was changed due to KIP-57 in new message format.
  • Minor refactorings

Big thanks to @fabregas for the hard work on this release (PR #60)

Minor fixes

18 Oct 08:33
Compare
Choose a tag to compare
  • Fixed bug with infinite loop on heartbeats with autocommit=True. #44
  • Bumped python-kafka to version 1.1.1
  • Fixed docker test runner with multiple interfaces
  • Minor documentation fixes

v0.1.2

30 Apr 12:31
Compare
Choose a tag to compare
  • Added Python3.5 usage example to docs
  • Don't raise retryable exceptions in 3.5's async for iterator
  • Fix Cancellation issue with producer's send_and_wait method