[Feature] Rolling/Windowing Functionality

[jaychia]

Summary

Many timeseries-type datasets benefit from a rolling/windowing API, where aggregations can be performed based on a specified window.

This feature lacks a proposed API and implementation, but please upvote, comment and share your use-cases if you would like to see this in Daft.

[marcosmartinezfco]

I would love to see some sort of functionality for sliding windows or timeseries to calculate things out of the box. I feel like the finance industry can benefit quite a lot from delta lake and daft but tasks such as calculating OHLC can be a bit more complex than other DataFrame libraries.

Maybe it's me that I'm a relatively new user, but I found it extremely hard to calculate the opening price and close price for a time series since there's no methods to get the first or last element when aggregating data, just min and max. Where as with pandas you simply resample the data on the timestamp and aggregate data.

[samster25]

@marcosmartinezfco Thanks for commenting here! So for discrete windows, we should be able to leverage this expression which will truncate a timestamp to days which you then can run a groupby. However today we don't have a concept of ordering for grouping which is why we can't do first or last.

Do you have an example of what the query would look like in SQL or pandas?

[marcosmartinezfco]

# pandas code
current_window_df.resample("1min")
                    .agg(
                        Open=("MDEntryPx", "first"),
                        High=("MDEntryPx", "max"),
                        Low=("MDEntryPx", "min"),
                        Close=("MDEntryPx", "last"),
                        Volume=("MDEntrySize", "sum"),
                    )
                    .dropna()
                    .reset_index()

The current window df is just a time stream of market updates where you have the price and the size basically.