feat(processing_engine): Runtime and Write-back improvements#25672
Merged
jacksonrnewhouse merged 1 commit intomainfrom Dec 18, 2024
Merged
feat(processing_engine): Runtime and Write-back improvements#25672jacksonrnewhouse merged 1 commit intomainfrom
jacksonrnewhouse merged 1 commit intomainfrom
Conversation
a24353c to
4e9c322
Compare
pauldix
approved these changes
Dec 17, 2024
Member
pauldix
left a comment
There was a problem hiding this comment.
A couple of small comments, but otherwise LGTM
* Move processing engine invocation to a seperate tokio task. * Support writing back line protocol from python via insert_line_protocol(). * Update structs to work with bincode.
4e9c322 to
354b185
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This makes two major changes, running plugins in their own tasks and enabling writing back line protocol to the WriteBuffer.
Running Plugins as Separate Tasks
Each plugin now runs in its own task, started either when created through the API or when the system is started. It is connected to the rest of the system through two mechanisms:
QueryableBufferand whose receiver it polls from.Right now the only message being sent is
Arc<WalContents>, but I expect the PluginEvent enum to be expanded both for other triggering events and control messages, e.g. Shutdown, Status, etc.Write-Back Line Protocol
The running pythons have a reference to the WriteBuffer, and this PR also adds a simple receiver for python to write line protocols to with the
insert_line_protocolcall. Everything is buffered per PluginEvent, and data is copied several times, so there is a lot of room for improvement.Example REST calls
Creating a Plugin:
Creating a Trigger: