Airflow 2.10.X and standalone OpenTelemetry Collector integration

[pavelekshin]

I'm trying to integrate Airflow 2.10.5 with a standalone OpenTelemetry Collector version 0.70, following the suggested setup script https://github.com/apache/airflow/blob/2.10.2/scripts/ci/docker-compose/integration-otel.yml. However, I'm encountering a 400 Bad Request error. The OpenTelemetry Collector logs don't show any entries related to the HTTP request. Any ideas on what might be going wrong or how to debug this further?

Airflow side:
[Metric Exporter] Connecting to OpenTelemetry Collector at http://otel-collector:4318/v1/metrics
[2026-03-17T10:16:03.577+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request
[2026-03-17T10:16:03.698+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request
[2026-03-17T10:16:08.585+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request
[2026-03-17T10:16:08.708+0000] {__init__.py:198} ERROR - Failed to export span batch code: 400, reason: Bad Request

Otel side: nothing

I also try gRPC via 4317, but airflow not switch to gRPC protocol and continuously send HTTP requests

Airflow side:
[2026-03-17T09:06:14.263+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (1): otel-collector:4317
[2026-03-17T09:06:14.267+0000] {connectionpool.py:241} DEBUG - Starting new HTTP connection (2): otel-collector:4317
[2026-03-17T09:06:14.267+0000] {__init__.py:266} WARNING - Transient error ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) encountered while exporting metrics batch, retrying in 0.98s

Otel side:
Closing: connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"POST /v1/metrics HTTP/1.\""

This works and return 200, but nothing on otel logs

curl -v -X POST http://otel-collector:4318/v1/metrics      -H "Content-Type: application/json"      -d '{"resourceMetrics": [{"resource": {"attributes": [{"key": "service.name", "value": {"stringValue": "test"}}]}}]}'

I also tested with the latest OpenTelemetry Collector (v0.146). The error changed from 400 to 500 Internal Server Error, but the collector logs remain completely empty. No entries appear even with debug logging enabled.

I tried tune exporter settings on airflow side using protocol settings - https://opentelemetry.io/docs/languages/sdk-configuration/otlp-exporter/

OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf, etc

but without any success

I use following airflow ENV:

    - AIRFLOW__METRICS__OTEL_ON=True
    - AIRFLOW__METRICS__OTEL_HOST=otel-collector
    - AIRFLOW__METRICS__OTEL_PORT=4318
    - AIRFLOW__METRICS__OTEL_INTERVAL_MILLISECONDS=30000
    - AIRFLOW__TRACES__OTEL_ON=True
    - AIRFLOW__TRACES__OTEL_HOST=otel-collector
    - AIRFLOW__TRACES__OTEL_PORT=4318
    - AIRFLOW__TRACES__OTEL_DEBUGGING_ON=False
    - AIRFLOW__TRACES__OTEL_TASK_LOG_EVENT=True

and installed otel plugin via apache-airflow[otel]

opentelemetry-api                        1.40.0
opentelemetry-exporter-otlp              1.40.0
opentelemetry-exporter-otlp-proto-common 1.40.0
opentelemetry-exporter-otlp-proto-grpc   1.40.0
opentelemetry-exporter-otlp-proto-http   1.40.0
opentelemetry-exporter-prometheus        0.61b0
opentelemetry-proto                      1.40.0
opentelemetry-sdk                        1.40.0
opentelemetry-semantic-conventions       0.61b0

otel/prometheus setup

    otel-collector:
        image: otel/opentelemetry-collector-contrib:0.70.0
        command: [--config=/etc/otel-collector-config.yml]
        volumes:
            - ./otel-collector-config.yml:/etc/otel-collector-config.yml
        ports:
            - "24318:4318"
            - "28889:8889"

    prometheus:
        image: prom/prometheus
        user: "0"
        ports:
            - "29090:9090"
        volumes:
            - ./prometheus-config.yml:/etc/prometheus/prometheus.yml

otel config:

receivers:
 otlp:
   protocols:
     http:
       endpoint: 0.0.0.0:4318
     grpc:
       endpoint: 0.0.0.0:4317

processors:
 batch:

exporters:
 logging:
   verbosity: detailed
 prometheus:
   endpoint: 0.0.0.0:8889

service:
 telemetry:
   logs:
     level: debug
     development: true

 pipelines:
   metrics:
     receivers: [otlp]
     processors: [batch]
     exporters: [logging, prometheus]

   traces:
     receivers: [otlp]
     processors: [batch]
     exporters: [logging]

[pavelekshin]

I get packet capture, and it's seems like airflow send binary protobuf body with application/json header, and reicive 400 bad request on this body.

it's should be application/x-protobuf, but content type is hardcoded

https://github.com/apache/airflow/blob/2.10.5/airflow/metrics/otel_logger.py#L410

POST /v1/metrics HTTP/1.1
Host: otel-collector:4318
User-Agent: OTel-OTLP-Exporter-Python/1.40.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: application/json
Content-Length: 3275


..
..
"
.telemetry.sdk.language..
.python
%
.telemetry.sdk.name..

opentelemetry
!
.telemetry.sdk.version..
.1.40.0
.
	host.name..
.96fb917f9fba
.
.service.name.	
.Airflow...
.
.airflow.metrics.otel_logger.6
.airflow.job_start:!
..	&.NK....z..Ag...1.............X
3airflow.dag_processing.file_path_queue_update_count:!
..,/.WK....z..Ag...1..............

 airflow.dag_processing.processes:.	
~.<..WK....z..Ag...1........:N
	file_path.A
?/usr/local/ECO/airflow/dags/maintenance/temp_cleanup_logs.py:.
.action..
.start
y..ZRXK....z..Ag...1........:I
	file_path.<
:/usr/local/ECO/airflow/dags/maintenance/cleanup_logs.py:.
.action..
.start
t..=.YK....z..Ag...1........:D
	file_path.7
5/usr/local/ECO/airflow/dags/maintenance/migrate.py:.
.action..
.start
.....gZK....z..Ag...1........:Q
	file_path.D
B/usr/local/ECO/airflow/dags/maintenance/cleanup_target_group.py:.
.action..
.start
..vw.]K....z..Ag...1........:N
	file_path.A
?/usr/local/ECO/airflow/dags/maintenance/temp_cleanup_logs.py:.
.action..
.finish
{..%.]K....z..Ag...1........:K
	file_path.>
</usr/local/ECO/airflow/dags/maintenance/load_variables.py:.
.action..
.start
z....^K....z..Ag...1........:I
	file_path.<
:/usr/local/ECO/airflow/dags/maintenance/cleanup_logs.py:.
.action..
.finish
u..]R^K....z..Ag...1........:D
	file_path.7
5/usr/local/ECO/airflow/dags/maintenance/migrate.py:.
.action..
.finish
|.._.bK....z..Ag...1........:K
	file_path.>
</usr/local/ECO/airflow/dags/maintenance/load_variables.py:.
.action..
.finish
.....%bK....z..Ag...1........:Q
	file_path.D
B/usr/local/ECO/airflow/dags/maintenance/cleanup_target_group.py:.
.action..
.finish...6
.airflow.serde.load_serializers*.
..z..Ag...!.....oR?.C
+airflow.dag_processing.file_path_queue_size*.
..z..Ag...!.........W
?airflow.dag_processing.last_num_of_db_queries.temp_cleanup_logs*.
..z..Ag...1.........R
:airflow.dag_processing.last_num_of_db_queries.cleanup_logs*.
..z..Ag...1.........M
5airflow.dag_processing.last_num_of_db_queries.migrate*.
..z..Ag...1
........W
?airflow.dag_processing.last_num_of_db_queries.cleanup_target_gr*.
..z..Ag...1.........T
<airflow.dag_processing.last_num_of_db_queries.load_variables*.
..z..Ag...1
........N
6airflow.dag_processing.last_duration.temp_cleanup_logs*.
..z..Ag...!.J..h..?.^
$airflow.dag_processing.last_duration*6
4.z..Ag...!.J..h..?: 
	file_name..
.temp_cleanup_logs.I
1airflow.dag_processing.last_duration.cleanup_logs*.
..z..Ag...!J	..z..?.?
'airflow.dag_processing.total_parse_time*.
..z..Ag...!..`.b..?.+
.airflow.dagbag_size*.
..z..Ag...1.........<
$airflow.dag_processing.import_errors*.
..z..Ag...!.........D
,airflow.dag_processing.last_duration.migrate*.
..z..Ag...!.. ..=.?.K
3airflow.dag_processing.last_duration.load_variables*.
..z..Ag...!..!r.z.?.Q
9airflow.dag_processing.last_duration.cleanup_target_group*.
..z..Ag...![.a/...?.U
=airflow.dag_processing.last_run.seconds_ago.temp_cleanup_logs*.
..z..Ag...!.X..C-"@.P
8airflow.dag_processing.last_run.seconds_ago.cleanup_logs*.
..z..Ag...!...K.*"@.K
3airflow.dag_processing.last_run.seconds_ago.migrate*.
..z..Ag...!.U.q70"@.W
?airflow.dag_processing.last_run.seconds_ago.cleanup_target_grou*.
..z..Ag...!.f...&"@.R
:airflow.dag_processing.last_run.seconds_ago.load_variables*.
..z..Ag...!S..c$."@HTTP/1.1 400 Bad Request
Content-Type: application/json
Date: Tue, 17 Mar 2026 15:08:09 GMT
Content-Length: 276

{"code":3,"message":"ReadObjectCB: expect { or n, but found \ufffd, error found in #2 byte of ...|\n\ufffd\u0019\n\ufffd\u0001\n\"\n\u0016te|..., bigger context ...|\n\ufffd\u0019\n\ufffd\u0001\n\"\n\u0016telemetry.sdk.language\u0012\u0008\n\u0006python\n%\n\u0012teleme|..."}POST /v1/metrics HTTP/1.1
Host: otel-collector:4318
User-Agent: OTel-OTLP-Exporter-Python/1.40.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Type: application/json
Content-Length: 3275

[pavelekshin]

Tke root of problem is wrong version of installed opentelemetry-exporter on airflow, I don't use constraint for installation and installed lastest version, there is user defined header had top priority rather on suggested opentelemetry-exporter == 1.27.0 on constraints

But, seems like we not needed to hardcode conte-type on otel_logger.py in airflow, it's better to setup via OTEL_ settings.

        self._session.headers.update(self._headers)
        self._session.headers.update(_OTLP_HTTP_HEADERS)
        # let users override our defaults
        self._session.headers.update(self._headers)

opentelemetry-api                        1.40.0
opentelemetry-exporter-otlp              1.40.0
opentelemetry-exporter-otlp-proto-common 1.40.0
opentelemetry-exporter-otlp-proto-grpc   1.40.0
opentelemetry-exporter-otlp-proto-http   1.40.0
opentelemetry-exporter-prometheus        0.61b0
opentelemetry-proto                      1.40.0
opentelemetry-sdk                        1.40.0
opentelemetry-semantic-conventions       0.61b0