generated from mintlify/starter
-
Notifications
You must be signed in to change notification settings - Fork 10
Add make-series, mv-expand, parse-kv, parse-where, project-rename #389
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
manototh
wants to merge
7
commits into
main
Choose a base branch
from
mano/apl-sep1
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
7 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,168 @@ | ||
| --- | ||
| title: make-series | ||
| description: 'This page explains how to use the make-series operator in APL.' | ||
| --- | ||
|
|
||
| The `make-series` operator creates time series data by aggregating values over specified time bins. You use it to turn event-based data into evenly spaced intervals, which is useful for visualizing trends, comparing metrics over time, or performing anomaly detection. | ||
|
|
||
| You find this operator useful when you want to: | ||
|
|
||
| - Analyze trends in metrics such as request duration, error rates, or throughput. | ||
| - Prepare data for charting in dashboards where regular time intervals are required. | ||
| - Aggregate trace or log data into time buckets for performance monitoring or incident analysis. | ||
|
|
||
| ## For users of other query languages | ||
|
|
||
| If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="Splunk SPL users"> | ||
|
|
||
| In Splunk SPL, you often use the `timechart` command to create time series. In APL, you achieve the same result with the `make-series` operator, which lets you explicitly define the aggregation, time field, and binning interval. | ||
|
|
||
| <CodeGroup> | ||
| ```sql Splunk example | ||
| index=sample-http-logs | ||
| | timechart span=1m avg(req_duration_ms) | ||
| ```` | ||
|
|
||
| ```kusto APL equivalent | ||
| ['sample-http-logs'] | ||
| | make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| </Accordion> | ||
| <Accordion title="ANSI SQL users"> | ||
|
|
||
| In ANSI SQL, you typically use `GROUP BY` with windowing functions or generated series to build time-based aggregations. In APL, the `make-series` operator is the dedicated tool for generating continuous time series with defined intervals, which avoids the need for joins with a calendar table. | ||
|
|
||
| <CodeGroup> | ||
| ```sql SQL example | ||
| SELECT | ||
| time_bucket('1 minute', _time) AS minute, | ||
| AVG(req_duration_ms) AS avg_duration | ||
| FROM sample_http_logs | ||
| WHERE _time > NOW() - interval '1 hour' | ||
| GROUP BY minute | ||
| ORDER BY minute | ||
| ``` | ||
|
|
||
| ```kusto APL equivalent | ||
| ['sample-http-logs'] | ||
| | make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 1m | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| </Accordion> | ||
| </AccordionGroup> | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Syntax | ||
|
|
||
| ```kusto | ||
| make-series [Aggregation [, ...]] | ||
| [default = DefaultValue] | ||
| on TimeField | ||
| [in Range] | ||
| step StepSize | ||
| [by GroupingField [, ...]] | ||
| ``` | ||
|
|
||
| ### Parameters | ||
|
|
||
| | Parameter | Description | | ||
| | ---------------- | --------------------------------------------------------------------------------------------------------------- | | ||
| | `Aggregation` | One or more aggregation functions (for example, `avg()`, `count()`, `sum()`) to apply over each time bin. | | ||
| | `default` | A value to use when no records exist in a time bin. | | ||
| | `TimeField` | The field containing timestamps used for binning. | | ||
| | `Range` | An optional range expression specifying the start and end of the series (for example, `from ago(1h) to now()`). | | ||
| | `StepSize` | The size of each time bin (for example, `1m`, `5m`, `1h`). | | ||
| | `GroupingField` | Optional fields to split the series by, producing multiple series in parallel. | | ||
|
|
||
| ### Returns | ||
|
|
||
| The operator returns a table where each row represents a group (if specified), and each aggregation function produces an array of values aligned with the generated time bins. | ||
|
|
||
| ## Use case examples | ||
|
|
||
| <Tabs> | ||
| <Tab title="Log analysis"> | ||
|
|
||
| You want to analyze how average request duration evolves over time, binned into 5-minute intervals. | ||
|
|
||
| **Query** | ||
|
|
||
| ```kusto | ||
| ['sample-http-logs'] | ||
| | make-series avg(req_duration_ms) default=0 on _time from ago(1h) to now() step 5m | ||
| ``` | ||
|
|
||
| [Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20make-series%20avg(req_duration_ms)%20default%3D0%20on%20_time%20from%20ago(1h)%20to%20now()%20step%205m%22%7D) | ||
|
|
||
| **Output** | ||
|
|
||
| | avg_req_duration_ms | | ||
| | ---------------------------- | | ||
| | [123, 98, 110, 105, 130...] | | ||
|
|
||
| The query produces a time series of average request durations across the last hour, grouped into 5-minute intervals. | ||
|
|
||
| </Tab> | ||
| <Tab title="OpenTelemetry traces"> | ||
|
|
||
| You want to monitor average span duration per service, binned into 10-minute intervals. | ||
|
|
||
| **Query** | ||
|
|
||
| ```kusto | ||
| ['otel-demo-traces'] | ||
| | make-series avg(duration) default=0 on _time from ago(2h) to now() step 10m by ['service.name'] | ||
| ``` | ||
|
|
||
| [Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'otel-demo-traces'%5D%20%7C%20make-series%20avg(duration)%20default%3D0%20on%20_time%20from%20ago(2h)%20to%20now()%20step%2010m%20by%20%5B'service.name'%5D%22%7D) | ||
|
|
||
| **Output** | ||
|
|
||
| | service.name | avg_duration | | ||
| | --------------- | ------------------------------ | | ||
| | frontend | [20ms, 18ms, 22ms, 19ms, ...] | | ||
| | checkout | [35ms, 40ms, 33ms, 37ms, ...] | | ||
|
|
||
| The query builds parallel time series for each service, showing average span duration trends. | ||
|
|
||
| </Tab> | ||
| <Tab title="Security logs"> | ||
|
|
||
| You want to analyze the rate of HTTP 500 errors in your logs per minute. | ||
|
|
||
| **Query** | ||
|
|
||
| ```kusto | ||
| ['sample-http-logs'] | ||
| | where status == '500' | ||
| | make-series count() default=0 on _time from ago(30m) to now() step 1m | ||
| ``` | ||
|
|
||
| [Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20where%20status%20%3D%3D%20'500'%20%7C%20make-series%20count()%20default%3D0%20on%20_time%20from%20ago(30m)%20to%20now()%20step%201m%22%7D) | ||
|
|
||
| **Output** | ||
|
|
||
| | count_status_401 | | ||
| | ------------------------ | | ||
| | [0, 1, 0, 2, 1, 0, ...] | | ||
|
|
||
| The query generates a time series of HTTP 500 error counts, grouped into 1-minute bins. | ||
|
|
||
| </Tab> | ||
| </Tabs> | ||
|
|
||
| ## List of related operators | ||
|
|
||
| - [extend](/apl/tabular-operators/extend-operator): Creates new calculated fields, often as preparation before `make-series`. Use `extend` when you want to preprocess data for time series analysis. | ||
| - [mv-expand](/apl/tabular-operators/mv-expand): Expands arrays into multiple rows. Use `mv-expand` to work with the arrays returned by `make-series`. | ||
| - [summarize](/apl/tabular-operators/summarize-operator): Aggregates rows into groups but does not generate continuous time bins. Use `summarize` when you want flexible grouping without forcing evenly spaced intervals. | ||
| - [top](/apl/tabular-operators/top-operator): Returns the top rows by a specified expression, not time series. Use `top` when you want to focus on the most significant values instead of trends over time. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
| title: mv-expand | ||
| description: 'This page explains how to use the mv-expand operator in APL.' | ||
| --- | ||
|
|
||
| The `mv-expand` operator expands dynamic arrays and property bags into multiple rows. Each element of the array or each property of the bag becomes its own row, while other columns are duplicated. | ||
|
|
||
| You use `mv-expand` when you want to analyze or filter individual values inside arrays or objects. This is especially useful when working with logs that include lists of values, OpenTelemetry traces that contain arrays of spans, or security events that group multiple attributes into one field. | ||
|
|
||
| ## For users of other query languages | ||
|
|
||
| If you come from other query languages, this section explains how to adjust your existing queries to achieve the same results in APL. | ||
|
|
||
| <AccordionGroup> | ||
| <Accordion title="Splunk SPL users"> | ||
|
|
||
| In Splunk SPL, the `mvexpand` command expands multi-value fields into separate events. The APL `mv-expand` operator works in a very similar way, splitting array values into individual rows. The main difference is that APL explicitly works with dynamic arrays or property bags, while Splunk handles multi-value fields implicitly. | ||
|
|
||
| <CodeGroup> | ||
| ```sql Splunk example | ||
| ... | mvexpand request_uri | ||
| ```` | ||
|
|
||
| ```kusto APL equivalent | ||
| ['sample-http-logs'] | ||
| | mv-expand uri | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| </Accordion> | ||
| <Accordion title="ANSI SQL users"> | ||
|
|
||
| In ANSI SQL, you use `CROSS JOIN UNNEST` or `CROSS APPLY` to flatten arrays into rows. In APL, `mv-expand` provides a simpler and more direct way to achieve the same result. | ||
|
|
||
| <CodeGroup> | ||
| ```sql SQL example | ||
| SELECT id, value | ||
| FROM logs | ||
| CROSS JOIN UNNEST(request_uris) AS t(value) | ||
| ``` | ||
|
|
||
| ```kusto APL equivalent | ||
| ['sample-http-logs'] | ||
| | mv-expand uri | ||
| ``` | ||
|
|
||
| </CodeGroup> | ||
|
|
||
| </Accordion> | ||
| </AccordionGroup> | ||
|
|
||
| ## Usage | ||
|
|
||
| ### Syntax | ||
|
|
||
| ```kusto | ||
| mv-expand [kind=(bag|array)] [with_itemindex=IndexFieldName] FieldName [to typeof(Typename)] [limit Rowlimit] | ||
| ``` | ||
|
|
||
| ### Parameters | ||
|
|
||
| | Parameter | Description | | ||
| | -------------------------------- | -------------------------------------------------------------------------------------- | | ||
| | `kind` | Optional. Specifies whether the column is a bag (object) or an array. Defaults to `array`. | | ||
| | `with_itemindex=IndexFieldName` | Optional. Outputs an additional column with the zero-based index of the expanded item. | | ||
| | `FieldName` | Required. The name of the column that contains an array or object to expand. | | ||
| | `to typeof(Typename)` | Optional. Converts each expanded element to the specified type. | | ||
| | `limit Rowlimit` | Optional. Limits the number of expanded rows per record. | | ||
|
|
||
| ### Returns | ||
|
|
||
| The operator returns a table where each element of the expanded array or each property of the expanded object is placed in its own row. Other columns are duplicated for each expanded row. | ||
|
|
||
| ## Use case example | ||
|
|
||
| When analyzing logs, some values can be stored as arrays. You can use `mv-expand` to expand them into individual rows for easier filtering. | ||
|
|
||
| **Query** | ||
|
|
||
| ```kusto | ||
| ['sample-http-logs'] | ||
| | limit 100 | ||
| | mv-expand territories | ||
| | summarize count = count() by territory_name = tostring(territories) | ||
| ``` | ||
|
|
||
| [Run in Playground](https://play.axiom.co/axiom-play-qf1k/query?initForm=%7B%22apl%22%3A%22%5B'sample-http-logs'%5D%20%7C%20limit%20100%20%7C%20mv-expand%20territories%20%7C%20summarize%20count%20%3D%20count()%20by%20territory_name%20%3D%20tostring(territories)%22%7D) | ||
|
|
||
| **Output** | ||
|
|
||
| | territory_name | count | | ||
| | ---------------- | ------- | | ||
| | United States | 67 | | ||
| | India | 22 | | ||
| | Japan | 12 | | ||
|
|
||
| This query expands the `territories` array into rows and counts the most frequent territories. | ||
|
|
||
| ## List of related operators | ||
|
|
||
| - [project](/apl/tabular-operators/project-operator): Selects or computes columns. Use it when you want to reshape data, not expand arrays. | ||
| - [summarize](/apl/tabular-operators/summarize-operator): Aggregates data across rows. Use it after expanding arrays to compute statistics. | ||
| - [top](/apl/tabular-operators/top-operator): Returns the top N rows by expression. Use it after expansion to find the most frequent values. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
hum no, this is describing a summarize operator. They seem similiar for sure! but different.
you can use this query to play about with the difference between the two: https://play.axiom.co/axiom-play-qf1k/query?qid=UiH3wAA3IfX-t29ng0&relative=1
the summarize will produce individual rows for every time bucket, the make-series will encode the whole thing into a single array on the avg_metric and timestamp field
the idea with make-series is that you can turn your events into array-based time series data, manipulate that array in some way (using the series_ functions usually), then use mv-exapand to turn it back into row-based time series data.
so it's not about visualizing trends or comparing metrics, but rather it provides a way to maniuplate aggregated, time-series data.
it's difficult to wrap your head around i know, it's much easier if you just play with it.
Here's another example of using make-series to build array based time-series data, then calling a series function on that array to make it a smoothed rolling average value, then using mv-expand to turn it back into row-based time series data for graphing
https://play.axiom.co/axiom-play-qf1k/query?qid=DJ19Fz8dxAH-t29ou0&relative=1