[connector/datadog] Missing tracing application metrics data in Datadog after upgrading to v0.107 or higher

[oddgrd]

Component(s)

connector/datadog

What happened?

Description

After upgrading opentelemetry-collector-contrib from v0.103.0 to v0.108.0, we realized that we were missing tracing application metrics in Datadog. I couldn't identify any clear breaking change from the release docs, so I:

• Downgraded the version to v0.107.0, but I was still not receiving metrics data.
• I downgraded to v0.106.1, and observed that we were receiving tracing application metrics again.
• I then took a closer look at the v0.107.0 release notes, and noticed that the connect.datadogconnector.NativeIngest feature was now enabled by default in [connector/datadog] Move feature gate connector.datadogconnector.NativeIngest to beta #34549.
• After disabling this feature, I started seeing tracing application metrics in Datadog again, while still on v0.107.0.
• I then tried updating to v0.110.0, in case there was some other change that fixed my issue. I still saw tracing metrics data with the feature disabled.
• I then deployed v0.110.0 with the feature enabled, and saw that the metrics data was absent again, like it was on v0.107 and v0.108.

This is my first issue in this repository, so let me know if anything is missing, and I'll be happy to add it!

Steps to Reproduce

• Upgrade to v0.107.0 or v0.110.0 and see if you stop receiving data for your tracing application metrics in Datadog.
• If you don't receive metrics data, try disabling the feature with --feature-gates=-connector.datadogconnector.NativeIngest.
• See if you are receiving tracing application metrics data in Datadog again.

Expected Result

I should continue to receive tracing applications metric data in Datadog after upgrading to v0.107.0.

Actual Result

I stopped receiving data for my existing metrics in Datadog.

Collector version

v0.107.0, v0.108.0, v0.110.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04")
Compiler(if manually compiled): (e.g., "go 14.2")

I experienced the issue with the docker.io/otel/opentelemetry-collector-contrib:0.110.0 image.

OpenTelemetry Collector configuration

---
receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: otel-collector
          scrape_interval: 10s
          static_configs:
          - targets:
              - 127.0.0.1:8888
        - job_name: component-metrics
          metrics_path: "/metrics"
          scrape_timeout: 5s
          scrape_interval: 10s
          static_configs:
            - targets:
                - 127.0.0.1:9090

processors:
  batch:
    timeout: 10s
    send_batch_size: 10
    send_batch_max_size: 100
  attributes:
    actions:
      - key: env
        value: ${env:ENV}
        action: insert
  resource:
    attributes:
      - key: deployment.environment
        action: upsert
        value: ${env:ENV}

# Needed for DD trace metrics generation, see https://docs.datadoghq.com/opentelemetry/guide/migration/
connectors:
  datadog/connector:
    traces:
      span_name_as_resource_name: true

exporters:
  datadog/exporter:
    api:
      site: datadoghq.eu
      key: ${env:DD_API_KEY}
    traces:
      trace_buffer: 25
      span_name_as_resource_name: true
    host_metadata:
      enabled: false

extensions:
  health_check:
service:
  extensions:
    - health_check
  pipelines:
    traces:
      receivers:
        - otlp
      processors:
        - batch
        - attributes
      exporters:
        - datadog/exporter
        - datadog/connector
    logs:
      receivers:
        - otlp
      processors:
        - batch
        - attributes
      exporters:
        - datadog/exporter
    metrics:
      receivers:
        - otlp
        - prometheus
        - datadog/connector
      processors:
        - batch
        - attributes
      exporters:
        - datadog/exporter

Log output

There was no change in collector logs after the update.

Additional context

No response

[github-actions]

Pinging code owners:

• connector/datadog: @mx-psi @dineshg13 @ankitpatel96

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[songy23]

Hi can you file a ticket via https://datadoghq.com/support/ with some some output from fileexporter or debugexporter?

[oddgrd]

Hi can you file a ticket via https://datadoghq.com/support/ with some some output from fileexporter or debugexporter?

Sure! I'll try to make time for it next week.

[rubencc]

Hi, I have the same situation.

Is there any way to solve this?

Thanks.

[oddgrd]

Hi, I have the same situation.

Is there any way to solve this?

Thanks.

I was able to work around it by disabling the new feature. See step two in the reproduction part of the original ticket. I didn't get around to following up on this with Datadog support, so I'm not sure if there is a better solution. And I'm on holiday now, so I won't be able to for a while.

[songy23]

This is the same as #36272 (comment)

@oddgrd if you can remove attributes processor and replace it with the resource processor in your pipelines, this should be fixed.

[philipcwhite]

Not sure why this is closed as it is still broken. I upgraded from 104 to 114 and can't send data to datadog either. Using the flag does not fix the issue. I opened a support case with them but as of now their exporter is broken.