Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: (metrics) integrate OTeL in duties and eth packages #1852

Open
wants to merge 12 commits into
base: stage
Choose a base branch
from

Conversation

oleg-ssvlabs
Copy link
Contributor

@oleg-ssvlabs oleg-ssvlabs commented Nov 13, 2024

Background
The observability team is currently working on a metrics refactor to address various challenges with the current implementation. As part of this refactor, Prometheus SDKs are being replaced with OTeL SDKs, which are industry-leading in observability and provide an abstraction layer for various observability "backends". Additionally, OTeL SDKs will be used to implement Traces. This aspect should be taken into consideration.

Implementation

  1. Central observability configurator was added (new root pkg observability). It looks simple, although it wasn't simple to decide what should "configurator" include mainly because OTeL supports two way of provider configuration - SDKs and environment variables and we want SSV Node users to be able to configure observability via native OTeL env vars. Some research showed that the vision of OTeL is "SDK first" meaning there are many features that aren't possible to configure via environment variables, yet possible via SDKs.However, the long-term aim is not to wrap OTeL environment variables into custom environment variables. Just FYI.

  2. Multiple components were instrumented, including the Execution Client, Event Syncer, and Duty Scheduler. The following metrics were introduced (either new or replacements for old ones) along with their attributes:

  • ssv.el.latency.duration (ssv.el.addr attributre)

  • ssv.el.syncing.distance (ssv.el.addr attributre)

  • ssv.el.syncing.status (ssv.el.addr, ssv.el.status attributre)

  • ssv.el.syncing.last_processed_block (ssv.el.addr attributre)

  • ssv.event_syncer.handler.events_processed (ssv.event_syncer.event_name attributre)

  • ssv.event_syncer.handler.events_failed (ssv.event_syncer.event_name attribute attributre)

  • ssv.event_syncer.handler.last_processed_block

  • ssv.operator.duty_scheduler.slot_ticker_delay.duration

  • ssv.operator.duty_scheduler.executions (ssv.duty.type attributre)

  • ssv.operator.duty_scheduler.committee_executions (ssv.runner.role attributre)

  1. Conventions.md which describes OTeL metric related conventions (mainly naming). Will continuously be updated. Conventions (are described in the document) are based on OTeL original conventions. Although, in some cases OTeL's ambiguity was tightened.

@oleg-ssvlabs oleg-ssvlabs force-pushed the otel-metrics branch 2 times, most recently from bb2739f to 0c69041 Compare November 13, 2024 15:15
@oleg-ssvlabs oleg-ssvlabs changed the base branch from main to stage November 13, 2024 15:15
@oleg-ssvlabs oleg-ssvlabs changed the title WIP: OTeL metrics instrumentaion OTeL metrics instrumentation - Duty Scheduler / Event Handler (Event Syncer) Nov 13, 2024
@oleg-ssvlabs oleg-ssvlabs force-pushed the otel-metrics branch 7 times, most recently from ec02308 to 46a4f2b Compare November 14, 2024 14:19
@nkryuchkov nkryuchkov changed the title OTeL metrics instrumentation - Duty Scheduler / Event Handler (Event Syncer) feat: (metrics) integrate OTeL in duties and eth packages Nov 14, 2024
@zktaiga zktaiga removed the request for review from nkryuchkov November 15, 2024 10:26
@zktaiga zktaiga removed the request for review from iurii-ssv November 15, 2024 10:26
@oleg-ssvlabs oleg-ssvlabs force-pushed the otel-metrics branch 2 times, most recently from 564b43f to 5cb528b Compare November 15, 2024 14:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant