Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support large object streaming #3061

Open
wants to merge 9 commits into
base: main
Choose a base branch
from

Conversation

yanchengnv
Copy link
Collaborator

Fixes # .

Description

This PR implements the following features:

  • A general-purpose object streamer.
  • A file streamer implemented with object streamer.

The object streaming solution works as follows:

  • On the sender side, a generator object is provided that generates objects to be streamed to the targets.
  • On the receiver side, a processor factory object is registered on channel/topic. When a stream object is received on the specified channel/topic, the factory object is called to get a new processor, which is used to process objects on the same stream.
  • Stream objects are defined as Shareable objects. This makes it possible to pass them through Filters before sending (in future PR).
  • Stream objects are sent through aux messages.

For the application to use the object streamer:

  • The application must first register processing information on the receiving side by calling engine.register_stream_processing() on certain channel and topic, and provide processor factory and optionally a stream_done callback to be called when the streaming is completed.
  • On the sending side, the app calls the engine.stream_shareables() to stream objects. Generator object must be provided, which generates an object to be streamed when called. The generator object is also called to process replies from receivers for each object sent.

Types of changes

  • Non-breaking change (fix or new feature that would not break existing functionality).
  • Breaking change (fix or new feature that would cause existing functionality to change).
  • New tests added to cover the changes.
  • Quick tests passed locally by running ./runtest.sh.
  • In-line docstrings updated.
  • Documentation updated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant