Error in alloy-logs when using OTLP destination.

[cxk314]

Hi,
We are getting following errors in alloy-logs when configured to send with OTLP destination.

ts=2024-12-02T14:38:04.932002269Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type \"text/html\"" interval=2.94364643s
ts=2024-12-02T14:38:04.93206268Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type \"text/html\"" interval=6.872794584s
ts=2024-12-02T14:38:04.940434736Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 502 (Bad Gateway); transport: received unexpected content-type \"text/html\"" interval=4.517639474s
ts=2024-12-02T14:38:07.88774863Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type \"text/html\"" interval=6.169007857s
ts=2024-12-02T15:28:41.417358175Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type \"text/html\"" interval=7.184971869s
ts=2024-12-02T15:28:43.417816325Z level=info msg="Exporting failed. Will retry the request after interval." component_path=/ component_id=otelcol.exporter.otlp.otlp error="rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 503 (Service Unavailable); transport: received unexpected content-type \"text/html\"" interval=5.44345728s

Configuration:

cluster:
  name: my-cluster

destinations:
  - name: otlp
    type: otlp
    url: https://np.np-shared..com
...
alloy-logs:
  enabled: true
  alloy:
      enableReporting: false
  controller:
    podAnnotations: {kubernetes.azure.com/set-kube-service-host-fqdn: "true"}

[petewall]

To me, this looks like the url (i.e. https://np.np-shared..com/ in your snippet) is not the true otlp endpoint. It's responding with HTML, rather than with the otlp grpc protocol. Make sure the full path to the otlp endpoint is correct and also check if you need to use grpc protocol: otlp or http protocol: http

[cxk314]

The error was on the destination side. Batches were too big, after reducing batch size it started to work:

destinations:
  - name: otlp
    type: otlp
    url: https://np-grpc.np-shared.com
    processors:
      batch:
        size: 2000
        maxSize: 2000
      k8sattributes: