Skip to content

Conversation

@ashwanthgoli
Copy link
Contributor

@ashwanthgoli ashwanthgoli commented Dec 3, 2025

What this PR does / why we need it:

  1. Adds a log exporter for xcap to summarise observations. Which observations to summarise and export is currently hard-coded, it can be made configurable if needed as the collector api is composable.
  2. Exports capture as stats.Result which gets used in query response. Only summary is populated, we can also set dataobj stats but i am not sure if its important as we will using the above summary line for monitoring the new engine.
  3. Removes stats collection using logqlmodel/stats for the new engine
  4. Move all stats definitions to a central place to avoid circular imports

example:

level=info ts=2025-12-02T14:45:08.324964585Z caller=exporter.go:126 component=query-engine <--REDACTED--> logs_dataset_bucket_get_range=2 logs_dataset_pages_cache_hit=3 logs_dataset_pages_download_duration_ns=47.412432ms logs_dataset_pages_download_requests=1 logs_dataset_pages_scanned=4 logs_dataset_primary_pages_compressed_bytes="33 kB" logs_dataset_primary_pages_downloaded=2 logs_dataset_primary_rows_read=5487 logs_dataset_primary_rows_read_bytes="58 kB" logs_dataset_read_calls=4 logs_dataset_rows_after_pruning=5487 logs_dataset_rows_max=5487 logs_dataset_secondary_pages_compressed_bytes="45 kB" logs_dataset_secondary_pages_downloaded=7 logs_dataset_secondary_rows_read=39 logs_dataset_secondary_rows_read_bytes="40 kB" metastore_bucket_attributes=1 metastore_bucket_get=1 metastore_bucket_get_range=7 metastore_index_objects=1 metastore_primary_pages_compressed_bytes="821 B" metastore_primary_pages_downloaded=9 metastore_resolved_sections=1 metastore_secondary_pages_compressed_bytes="1.5 kB" metastore_secondary_pages_downloaded=13 streams_bucket_get_range=2 streams_primary_pages_compressed_bytes="8 B" streams_primary_pages_downloaded=1 streams_secondary_pages_compressed_bytes="191 B" streams_secondary_pages_downloaded=4

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

collect.fromRegions("DataObjScan", true, "streamsView.init").
filter(
// object store calls
"bucket.get", "bucket.getrange", "bucket.attributes",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

observations names are hardcoded, I plan to move the stat definitions to a central place so we can directly refer to them.

@ashwanthgoli ashwanthgoli changed the base branch from main to xcap-read-stats December 4, 2025 07:35
@ashwanthgoli ashwanthgoli marked this pull request as ready for review December 4, 2025 09:16
@ashwanthgoli ashwanthgoli requested a review from a team as a code owner December 4, 2025 09:16
Copy link
Contributor

@benclive benclive left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One question on field naming but otherwise LGTM!

}
}

// Format duration values (keys ending with "duration_ns")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do these keys hold nanoseconds specifically? Prometheus-style metrics usually just use "seconds" as the unit so maybe we want to use that, rather than doing something different?
I believe .String() on a duration already formats it into something like "3.23s" so the nanoseconds piece would get ignored anyway

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

they do hold ns now, i can switch it to secs 👍

// Close the pipeline to calculate the stats.
pipeline.Close()

queueTime, _ := ctx.Value(httpreq.QueryQueueTimeHTTPHeader).(time.Duration)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this queue time accurate? Could we pass it into the ToStatsSummary already and resolve one of the TODOs?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i doubt it, in the code i see its being set by the old scheduler. I removed it intentionally to track it propery in a follow-up

Base automatically changed from xcap-read-stats to main December 4, 2025 13:05
@ashwanthgoli ashwanthgoli enabled auto-merge (squash) December 4, 2025 13:15
@ashwanthgoli ashwanthgoli merged commit 54f9723 into main Dec 4, 2025
92 of 124 checks passed
@ashwanthgoli ashwanthgoli deleted the xcap-log-summary branch December 4, 2025 13:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants