Skip to content

Releases: Bendabir/torch-kafka

Make records skippable by the dataset

12 Aug 14:28
Compare
Choose a tag to compare

This release allows user to skip some output of the _process method by returning None.

Easier consumer instanciation override

10 Jun 11:52
Compare
Choose a tag to compare

Minor correction to ease the override of consumer parameters when instanciated.

Python package

06 Nov 22:11
Compare
Choose a tag to compare

This release addresses the following items :

  • The module can now be installed as a regular package. You can use pip install git+https://github.com/bendabir/torch-kafka@1.1.0 to do so.
  • Fixed possible deadlocks when committing offsets using multiprocessing.

Minor fixes

20 Oct 13:21
Compare
Choose a tag to compare

This fixes some possible minor issues :

  • signal handler used to commit the workers is now reset to default handler once iteration is over.
  • auto_commit helper now works on all datasets and has no effect on datasets that are not instances of KafkaDataset.

Initial release

15 Oct 21:15
Compare
Choose a tag to compare

This first release provide a PyTorch Dataset for Kafka and a method to auto-commit the data that are generated by a PyTorch DataLoader. It supports both single-processing and multi-processing.

Supported system and versions are :

  • Linux (tested with Ubuntu 18.04);
  • Python 3.8+;
  • PyTorch 1.6+;
  • kafka-python 2.0+.