Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement count windows #736

Merged
merged 7 commits into from
Feb 5, 2025
Merged

implement count windows #736

merged 7 commits into from
Feb 5, 2025

Conversation

quentin-quix
Copy link
Contributor

@quentin-quix quentin-quix commented Feb 3, 2025

Implements:

  • sdf.tumbling_count_window
  • sdf.hopping_count_window
  • sdf.sliding_count_window

Those windows divide messages into fixed-size batch of messages. They allow performing stateful aggregations like sum, reduce, etc. on top of a fixed number of messages and emit results downstream.

app = Application()
sdf = app.dataframe(...)

sdf = (
    # Define a tumbling window of 10 messages
    sdf.tumbling_count_window(count=10)

    # Specify the aggregation function
    .sum()

    # Specify how the results should be emitted downstream.
    # "current()" will emit results as they come for each updated window,
    # possibly producing multiple messages per key-window pair
    # "final()" will emit windows only when they are closed and cannot
    # receive any updates anymore.
    .current()
)

@quentin-quix quentin-quix force-pushed the quent/couting-windows-part3 branch 3 times, most recently from f99d1f2 to 3e47900 Compare February 4, 2025 12:52
@quentin-quix quentin-quix force-pushed the quent/couting-windows-part3 branch from 3e47900 to 258f1c7 Compare February 4, 2025 14:47
@quentin-quix quentin-quix changed the base branch from quent/couting-windows-part2 to main February 4, 2025 14:48
@quentin-quix quentin-quix marked this pull request as ready for review February 4, 2025 14:49
@daniil-quix daniil-quix added the feature New feature or request label Feb 5, 2025
@quentin-quix quentin-quix force-pushed the quent/couting-windows-part3 branch from 8323eb9 to 143078d Compare February 5, 2025 11:19
@quentin-quix quentin-quix force-pushed the quent/couting-windows-part3 branch from 143078d to 57a7c0e Compare February 5, 2025 11:21
docs/windowing.md Outdated Show resolved Hide resolved
@quentin-quix quentin-quix merged commit 0bb8d15 into main Feb 5, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
Development

Successfully merging this pull request may close these issues.

2 participants