How to send traces to AWS X-Ray?

[Lawouach]

Hi all,

I'm running the demo in an AWS EKS cluster and I've enabled the AWS ADOT collector.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector-xray
spec:
  mode: deployment 
  resources:
    requests:
      cpu: "1"
    limits:
      cpu: "1"
  serviceAccount: adot-collector
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch/traces:
        timeout: 1s
        send_batch_size: 50

    exporters:
      awsxray:
        region: eu-central-1

    service:
      telemetry:
        logs:
          level: "debug"
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch/traces]
          exporters: [awsxray]

I have also configured the demo collector to use the ADOT collector as its exporter:

opentelemetry-collector:
  config:
    exporters:
      otlp/xray:
        endpoint: http://my-collector-xray-collector:4317
        tls:
          insecure: true

    service:
      pipelines:
        traces:
          exporters: [spanmetrics, otlp/xray]

I'm fairly certain traces are sent accordingly to the ADOT collector. But they never reach the X-Ray service. My feeling is that it's because there is no AWS x-ray-id generator enabled on the demo. Thus traces are ignored by AWS.

But short of forking the demo and amending all services to add such generator at the code level, I am not sure if that's actually possible via configuration.

Has anyone been successful with such configuration?

Thanks,

• Sylvain

[Lawouach]

Right, so I tried something different. I disabled the demo collector so that traces would be sent directly to the AWS ADOT collector like this:

default:
  envOverrides:
  - name: OTEL_COLLECTOR_NAME
    value: otel-collector-xray-collector

opentelemetry-collector:
  enabled: false

Traces show up on the collector but it complains about the "invalid" trace id (since it doesn't match the format expected by X-RAy:

debug	awsxrayexporter@v0.70.0/awsxray.go:105	Error translating span.	{"kind": "exporter", "data_type": "traces", "name": "awsxray", "error": "invalid xray traceid: 67a11ce91e02858d9a8e89c6aeece454"}

[Lawouach]

It seems indeed this is confirmed here https://aws-otel.github.io/docs/getting-started/adot-eks-add-on/config-xray

Note that segments or spans ingested by the OTLP receiver must be in X-Ray Segment format.

[puckpuck]

I expect the AWS x-ray exporter to handle everything required to get the data into the proper format. How does this exporter work with any other OpenTelemetry instrumented application?

[Lawouach]

On Python for instance, you would need to use a specific IDGenerator https://opentelemetry-python-contrib.readthedocs.io/en/latest/sdk-extension/aws/aws.html so traces are properly setup from the get go.

[puckpuck]

Ooof. I hoped the x-ray exporter detected this and fixed the traceIds on export, but clearly, it does not.

We don't want to start adding x-ray specific things to the demo code to support something that is vendor-specific. Can you raise this issue with the x-ray exporter for the OpenTelemetry collector? I assume someone from Amazon will want to have x-ray work in an OpenTelemetry world without requiring any special code for the instrumentation.

[arun-annamalai]

Amazon Xray now supports the Otel Trace ID. So you should not need to add x-ray specific things to the demo code.

Source: https://aws.amazon.com/about-aws/whats-new/2023/10/aws-x-ray-w3c-format-trace-ids-distributed-tracing/