Skip to content

[otel/kube-stack]: Add gateway collector #6444

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 40 commits into from
Jan 27, 2025
Merged
Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
67371a7
feat: move telemetry aggregation and forwarding to gateway
rogercoll Dec 27, 2024
9d84b4a
ci: use Elastic envs in gateway
rogercoll Dec 27, 2024
662557f
chore: add changelog entry
rogercoll Dec 27, 2024
36aeb8e
fix: format values file
rogercoll Dec 27, 2024
ca4b80f
feat: add apm loadbalancing
rogercoll Jan 3, 2025
f990086
chore: increase resource limits
rogercoll Jan 6, 2025
f0a8c5c
Merge branch 'add_gateway_collector_lb' into add_gateway_collector
rogercoll Jan 7, 2025
7d50d2a
revert resource limits increase
rogercoll Jan 7, 2025
f8a8581
Merge branch 'main' into add_gateway_collector
rogercoll Jan 7, 2025
5053e0b
chore: remove config warnings
rogercoll Jan 8, 2025
16247d6
Merge branch 'main' into add_gateway_collector
rogercoll Jan 8, 2025
238d6c3
docs: add Gateway collectors section
rogercoll Jan 8, 2025
b0fd91d
revert: enable daemonset storagechecks
rogercoll Jan 8, 2025
e633c52
Merge branch 'main' into add_gateway_collector
rogercoll Jan 8, 2025
8d859d6
rename metrics/otel pipeline and use signaltometrics
rogercoll Jan 9, 2025
1652f14
unify k8s and host metrics pipelines
rogercoll Jan 9, 2025
3051242
use default traceID as loadbalancing routing_key
rogercoll Jan 9, 2025
c5c52aa
chore: reuse k8s integration test helpers
rogercoll Jan 10, 2025
9dedcfe
format values with Helm linter
rogercoll Jan 10, 2025
5b3c4aa
Merge branch 'main' into add_gateway_collector
rogercoll Jan 13, 2025
3211ce7
replace loadbalancing in favor of headless otlp
rogercoll Jan 13, 2025
04bcbee
Update testing/integration/otel_helm_test.go
rogercoll Jan 14, 2025
dce455f
Update testing/integration/otel_helm_test.go
rogercoll Jan 14, 2025
ab64abd
rename k8s values options helper function
rogercoll Jan 14, 2025
9b0a6c4
Merge branch 'main' into add_gateway_collector
rogercoll Jan 14, 2025
090abfc
move process attributes remove processor to gateway
rogercoll Jan 15, 2025
af831c8
add batch processor for aggregation pipeline
rogercoll Jan 16, 2025
eebe965
Merge branch 'main' into add_gateway_collector
rogercoll Jan 17, 2025
d538b4b
enable compression for cluster otlp connections
rogercoll Jan 17, 2025
b4f8482
chore: remove elastic endpoint references
rogercoll Jan 17, 2025
b273771
Merge branch 'main' into add_gateway_collector
rogercoll Jan 20, 2025
ffa6620
fix: do not generate service's signals for non apm data
rogercoll Jan 21, 2025
9b1587d
Revert "fix: do not generate service's signals for non apm data"
rogercoll Jan 21, 2025
b563651
fix: set agent.name as edot-collector
rogercoll Jan 23, 2025
6179c9e
fix: enable daemon hostNetwork
rogercoll Jan 23, 2025
c98d3c6
Merge branch 'main' into add_gateway_collector
rogercoll Jan 23, 2025
7cdb994
set unknown as default signaltometrics agent.name resource attribute
rogercoll Jan 23, 2025
9825bad
Merge branch 'main' into add_gateway_collector
rogercoll Jan 23, 2025
c83cd8f
remove signaltometrics for metrics-only services
rogercoll Jan 24, 2025
47356d8
Merge branch 'main' into add_gateway_collector
rogercoll Jan 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Kind can be one of:
# - breaking-change: a change to previously-documented behavior
# - deprecation: functionality that is being removed in a later release
# - bug-fix: fixes a problem in a previous version
# - enhancement: extends functionality but does not break or fix existing behavior
# - feature: new functionality
# - known-issue: problems that we are aware of in a given version
# - security: impacts on the security of a product or a user’s deployment.
# - upgrade: important information for someone upgrading from a prior version
# - other: does not fit into any of the other categories
kind: feature

# Change summary; a 80ish characters long description of the change.
summary: add gateway collector in Helm kube-stack deployment

# Long description; in case the summary is not enough to describe the change
# this field accommodate a description without length limits.
# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
#description:

# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
component: "elastic-agent"

# PR URL; optional; the PR number that added the changeset.
# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
# Please provide it if you are adding a fragment for a different PR.
#pr: https://github.com/owner/repo/1234

# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
# If not present is automatically filled by the tooling with the issue linked to the PR number.
#issue: https://github.com/owner/repo/1234
15 changes: 13 additions & 2 deletions deploy/helm/edot-collector/kube-stack/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,26 @@ The DaemonSet collectors handle the following data:
- Logs: Utilizes [File Log Receiver receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/filelogreceiver#readme) to gather logs from all Pods running on the respective node.
- OTLP Traces: Utilizes [OTLP Receiver]( https://github.com/open-telemetry/opentelemetry-collector/blob/main/receiver/otlpreceiver#readme) which configures both HTTP and GRPC endpoints on the node to receive OTLP trace data.

### Deployment collector
### Deployment collectors

#### Cluster

The OpenTelemetry components deployed within a Deployment collector focus on gathering data at the cluster level rather than at individual nodes. A Deployment instance of the collector operates as a standalone (unlike DaemonSet collector instances, which are deployed on every node)

The Deployment collector handles the following data:
The Cluster Deployment collector handles the following data:

- Kubernetes Events: Monitors and collects events occurring across the entire Kubernetes cluster, utilizing [Kubernetes Objects Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sobjectsreceiver#readme).
- Cluster Metrics: Captures metrics that provide insights into the overall health and performance of the Kubernetes cluster, utilizing [Kubernetes Cluster Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/k8sclusterreceiver#readme).

#### Gateway

The OpenTelemetry components deployed within the `Gateway` Deployment collectors focus on processing and exporting OTLP data to Elasticsearch. Processing components:

- [Elastic Trace processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elastictraceprocessor): The processor enriches traces with elastic specific requirements. It uses opentelemetry-lib to perform the actual enrichments.
- [Elastic Infra Metrics processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/elasticinframetricsprocessor): The Elastic Infra Metrics Processor is used to bridge the gap between OTEL and Elastic Infra Metrics.
- [LSM interval processor](https://github.com/elastic/opentelemetry-collector-components/tree/main/processor/lsmintervalprocessor): [Interval processor](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/intervalprocessor) with db-backed persistence.
- [Signal to Metrics connector](https://github.com/elastic/opentelemetry-collector-components/tree/main/connector/signaltometricsconnector): Produces metrics from all signal types (traces, logs, or metrics).

### Auto-instrumentation

The Helm Chart is configured to enable zero-code instrumentation using the [Operator's Instrumentation resource](https://github.com/open-telemetry/opentelemetry-operator/?tab=readme-ov-file#opentelemetry-auto-instrumentation-injection) for the following programming languages:
Expand Down
Loading
Loading