Skip to content

Conversation

@2010YOUY01
Copy link
Contributor

Which issue does this PR close?

Part of #18116

Rationale for this change

The below configuration can be used to let EXPLAIN ANALYZE only show important high-level insights.

set datafusion.explain.analyze_level = summary;

This PR sets summary level metrics for the parquet data source:

summary level metrics for DataSourceExec with Parquet source

The remaining metrics are kept in the dev level. I'm not sure if the page level pruning metrics should also be included to the summary level, I'm open to suggestions for this, or any other metrics that should also be included.

While implementing this, I came up with a few ideas to further improve metrics tracking in the Parquet scanner. I’ve documented them in #18195

What changes are included in this PR?

Set the above metrics to summary analyze level

Are these changes tested?

UTs

Are there any user-facing changes?

No

@github-actions github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate labels Oct 21, 2025
@xudong963
Copy link
Member

page level pruning metrics

I'd like to see it under any context.

@2010YOUY01
Copy link
Contributor Author

page level pruning metrics

I'd like to see it under any context.

I see. I have added page pruning metrics to summary analyze level in 7a3fd2d

Copy link
Member

@xudong963 xudong963 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you @2010YOUY01

@2010YOUY01 2010YOUY01 added this pull request to the merge queue Oct 25, 2025
Merged via the queue into apache:main with commit f4a49b5 Oct 25, 2025
28 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

core Core DataFusion crate datasource Changes to the datasource crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants