Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Add PPL support to unnest arrays #644

Open
A-Gray-Cat opened this issue Sep 11, 2024 · 2 comments
Open

[FEATURE] Add PPL support to unnest arrays #644

A-Gray-Cat opened this issue Sep 11, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@A-Gray-Cat
Copy link

Is your feature request related to a problem?
Many log sources include arrays in a log line, and to efficiently extract and analyze these data, it would be very helpful to have a function in place to separate one log line that contains an n-element array to n log lines that contain one element from the array.

What solution would you like?
It's similar to the explode function in Spark SQL, and the expand function in Splunk.

What alternatives have you considered?
A clear and concise description of any alternative solutions or features you've considered.

Do you have any additional context?
Add any other context or screenshots about the feature request here.

@A-Gray-Cat A-Gray-Cat added enhancement New feature or request untriaged labels Sep 11, 2024
@YANG-DB
Copy link
Member

YANG-DB commented Sep 11, 2024

@A-Gray-Cat thanks for your request - can you please add some context here if possible for an example command syntax or other language examples for such functionality ?

@YANG-DB YANG-DB removed the untriaged label Sep 11, 2024
@A-Gray-Cat
Copy link
Author

It's similar to the explode function in spark:
explode(expr) Separates the elements of array `expr` into multiple rows, or the elements of map `expr` into multiple rows and columns. Unless specified otherwise, uses the default column name `col` for elements of the array or `key` and `value` for the elements of the map.

Link: https://spark.apache.org/docs/latest/sql-ref-functions-builtin.html#generator-functions

When using this in Spark SQL, it would be used with LATERAL VIEW at the same time:

SELECT r
FROM securitylake.amazon_security_lake_glue_db_us_east_1.amazon_security_lake_table_us_east_1_sh_findings_2_0 a
LATERAL VIEW EXPLODE(a.resources) as r

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants