Skip to content

Commit

Permalink
remove redundant reference to tail-sampling. It's already covered ear…
Browse files Browse the repository at this point in the history
…lier in the doc that these considerations should be made
  • Loading branch information
michael2893 authored and chalin committed Jun 8, 2024
1 parent ca2b41a commit 57400a8
Show file tree
Hide file tree
Showing 3 changed files with 51 additions and 79 deletions.
8 changes: 4 additions & 4 deletions content/en/docs/collector/deployment/multiple-collectors.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,10 @@ report on the same data can lead to data loss or, at least, degraded data
quality. In gateway collector deployments, applying this principle guards
against sending inconsistent data to the backend. All metric data streams within
OTLP must have a
[single writer](/docs/specs/otel/metrics/data-model/#single-writer).
In a system with multiple collectors, the single-writer principle is most
relevant for receivers that create their own metrics, such a pull-based scrapers
or a host metrics receiver.
[single writer](/docs/specs/otel/metrics/data-model/#single-writer). In a system
with multiple collectors, the single-writer principle is most relevant for
receivers that create their own metrics, such a pull-based scrapers or a host
metrics receiver.

### Deployment Considerations

Expand Down
114 changes: 39 additions & 75 deletions content/en/docs/collector/scaling.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,36 +389,6 @@ The Single-Writer Principle refers to employing a single logical writer for a
particular resource. When scaling collectors horizontally in a system it's
important to properly distinguish between targets using unique identities.

##### Tail-based Sampling

Partial or incomplete traces may be consequential for an implementation of
tail-sampling, as the goal is to capture all or most of the spans within the
trace in order to inform sampling decisions. When using the target allocator to
identify targets for scraping, you can also add labels.

```yaml
- job_name: 'otel-collector'
static_configs:
- targets: ['0.0.0.0:8888']
labels:
service: 'my-service'
```

You can also use
[relabel](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config)
to manage labeling on jobs.

```yaml
job_name: 'otel-collector'
static_configs:
- targets: ['0.0.0.0:8888']
labels:
service: 'my-service'
relabel_configs:
- source_labels: [service]
target_label: app
action: replace
```

##### Pull-based Scraping

Expand All @@ -429,52 +399,46 @@ lower, you may be able to rely on sharding based on namespace or workload alone.
As the system increases in complexity, consider adding custom labels related to
the application or service to better delineate the targets.

```yaml
scrape_configs:
- job_name: 'otel-collector-dev'
static_configs:
- targets: ['test-service:metrics-port']
relabel_configs:
- source_labels: ['__meta_kubernetes_namespace'] # Default prometheus label
target_label: 'namespace'
action: keep
```

If you have metadata defined for your deployment, you can use that to further
refine the target.

Example manifest:
Here is an example of how to configure target labels in a collector
configuration that uses a Prometheus receiver.

```yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: test-service
labels:
tier: frontend
```
# config.yaml example
receivers:
prometheus:
config:
scrape_configs:
- job_name: 'otel-collector-dev-test-service-frontend'
static_configs:
- targets: ['test-service:metrics-port']
relabel_configs:
- source_labels: ['__meta_kubernetes_namespace']
target_label: 'namespace'
action: keep
- source_labels: ['__meta_kubernetes_service_label_tier']
target_label: 'tier'
action: keep
- job_name: 'otel-collector-dev-test-service-backend'
static_configs:
- targets: ['test-service:metrics-port']
relabel_configs:
- source_labels: ['__meta_kubernetes_namespace']
target_label: 'namespace'
action: keep
- source_labels: ['__meta_kubernetes_service_label_tier']
target_label: 'tier'
action: keep
```yaml
scrape_configs:
- job_name: 'otel-collector-dev-test-service-frontend'
static_configs:
- targets: ['test-service:metrics-port']
relabel_configs:
- source_labels: ['__meta_kubernetes_namespace']
target_label: 'namespace'
action: keep
- source_labels: ['__meta_kubernetes_service_label_tier']
target_label: 'tier'
action: keep
- job_name: 'otel-collector-dev-test-service-backend'
static_configs:
- targets: ['test-service:metrics-port']
relabel_configs:
- source_labels: ['__meta_kubernetes_namespace']
target_label: 'namespace'
action: keep
- source_labels: ['__meta_kubernetes_service_label_tier']
target_label: 'tier'
action: keep
exporters:
otlp:
endpoint: my.sample:4317
tls:
insecure: true
service:
pipelines:
metrics:
receivers: [prometheus]
exporters: [otlp]
```
8 changes: 8 additions & 0 deletions content/en/docs/concepts/sampling/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,3 +137,11 @@ For the individual language specific implementations of the OpenTelemetry API &
SDK you will find support for sampling at the respective documentation pages:

{{% sampling-support-list " " %}}

### Single-Writer Principle

When scaling collectors for distributed tracing with tail sampling, it's important to consider
the Single-Writer Principle. This principle ensures a single logical writer makes sampling decisions for a
particular trace, avoiding conflicts that might arise from multiple collectors evaluating the same trace for
sampling. Techniques like using load balancing exporters in front of tail-sampling collectors can help achieve this.

0 comments on commit 57400a8

Please sign in to comment.