Works with debug, but not something like googlecloud

[iancote-ionq]

Hello! Thank you for working on this, I need something exactly like this.

When I try to export to googlecloud I'm getting errors like this:

2024-11-12T01:11:33.994Z	info	MetricsExporter	{"kind": "exporter", "data_type": "metrics", "name": "debug", "resource metrics": 1, "metrics": 6, "data points": 8}
2024-11-12T01:11:33.995Z	info	ResourceMetrics #0
Resource SchemaURL: 
Resource attributes:
     -> service.name: Str(test)
     -> service.instance.id: Str(more_test)
ScopeMetrics #0
ScopeMetrics SchemaURL: 
InstrumentationScope  
Metric #0
Descriptor:
     -> Name: ping.rtt
     -> Description: 
     -> Unit: ms
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:31.951675877 +0000 UTC
Value: 1.404978
NumberDataPoints #1
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:32.952326551 +0000 UTC
Value: 1.439537
NumberDataPoints #2
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.95269958 +0000 UTC
Value: 1.412412
Metric #1
Descriptor:
     -> Name: ping.rtt.min
     -> Description: 
     -> Unit: ms
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.994490837 +0000 UTC
Value: 1.404978
Metric #2
Descriptor:
     -> Name: ping.rtt.max
     -> Description: 
     -> Unit: ms
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.994490837 +0000 UTC
Value: 1.439537
Metric #3
Descriptor:
     -> Name: ping.rtt.avg
     -> Description: 
     -> Unit: ms
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.994490837 +0000 UTC
Value: 1.418976
Metric #4
Descriptor:
     -> Name: ping.rtt.stddev
     -> Description: 
     -> Unit: ms
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.994490837 +0000 UTC
Value: 0.014852
Metric #5
Descriptor:
     -> Name: ping.loss.ratio
     -> Description: 
     -> Unit: 
     -> DataType: Gauge
NumberDataPoints #0
Data point attributes:
     -> net.peer.ip: Str(8.8.8.8)
     -> net.peer.name: Str(8.8.8.8)
StartTimestamp: 1970-01-01 00:00:00 +0000 UTC
Timestamp: 2024-11-12 01:11:33.994490837 +0000 UTC
Value: 0.000000
	{"kind": "exporter", "data_type": "metrics", "name": "debug"}
2024-11-12T01:11:34.075Z	error	internal/queue_sender.go:92	Exporting failed. Dropping data.	{"kind": "exporter", "data_type": "metrics", "name": "googlecloud", "error": "rpc error: code = InvalidArgument desc = One or more TimeSeries could not be written: timeSeries[2]: Field timeSeries[2] had an invalid value: Duplicate TimeSeries encountered. Only one point can be written per TimeSeries per request.; timeSeries[1]: Field timeSeries[1] had an invalid value: Duplicate TimeSeries encountered. Only one point can be written per TimeSeries per request.\nerror details: name = Unknown  desc = total_point_count:8 success_point_count:6 errors:{status:{code:3} point_count:2}", "dropped_items": 8}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
	go.opentelemetry.io/collector/exporter@v0.111.0/exporterhelper/internal/queue_sender.go:92
go.opentelemetry.io/collector/exporter/internal/queue.(*boundedMemoryQueue[...]).Consume
	go.opentelemetry.io/collector/exporter@v0.111.0/internal/queue/bounded_memory_queue.go:52
go.opentelemetry.io/collector/exporter/internal/queue.(*Consumers[...]).Start.func1
	go.opentelemetry.io/collector/exporter@v0.111.0/internal/queue/consumers.go:43

If I set it to use something simple like the debug exporter, it seems fine.

I'm configuring otel like this (stripped down config for testing, the googlecloud part works fine for a number of other metrics and logs being sent now):

receivers:
  icmpcheck:
    collection_interval: 30s
    targets:
      - target: 8.8.8.8

processors:
  resource:
    attributes:
      - key: service.name
        value: test
        action: upsert
      - key: service.instance.id
        value: more_test
        action: upsert

exporters:
  googlecloud:
    log:
      default_log_name: opentelemetry.io/collector-exported-log

  debug:
    verbosity: detailed

service:
  pipelines:
    metrics/test:
      receivers:
        - icmpcheck
      processors:
        - resource
      exporters:
        - googlecloud
        - debug

The metrics are showing up in GCP as expected though, just with the log noise. Any ideas?

[thmshmm]

Might be that the googlecloud exporter uses the StartTimestamp which is unset and thus always the same. Could you maybe try with the attribute processor to copy the timestamp into starttimestamp and validate if that works?

[iancote-ionq]

Apologies for my ignorance, but how does one add a starttimestamp? I've found many ways to modify attributes, but timestamps seem to be above that level: Here's a json sample: {"resourceMetrics":[{"resource":{},"scopeMetrics":[{"scope":{},"metrics":[{"name":"ping.rtt","unit":"ms","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441629004135281","asDouble":1.334812},{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441630005306713","asDouble":2.241777},{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631004853906","asDouble":1.207437}]}},{"name":"ping.rtt.min","unit":"ms","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631055550689","asDouble":1.207437}]}},{"name":"ping.rtt.max","unit":"ms","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631055550689","asDouble":2.241777}]}},{"name":"ping.rtt.avg","unit":"ms","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631055550689","asDouble":1.594675}]}},{"name":"ping.rtt.stddev","unit":"ms","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631055550689","asDouble":0.460515}]}},{"name":"ping.loss.ratio","gauge":{"dataPoints":[{"attributes":[{"key":"net.peer.ip","value":{"stringValue":"8.8.8.8"}},{"key":" net.peer.name ","value":{"stringValue":"8.8.8.8"}}],"timeUnixNano":"1731441631055550689","asDouble":0}]}}]}]}]}

…

On Tue, Nov 12, 2024 at 8:51 AM Thomas Hamm ***@***.***> wrote: Might be that the googlecloud exporter uses the StartTimestamp which is unset and thus always the same. Could you maybe try with a processor to copy the timestamp into starttimestamp and validate if that works? — Reply to this email directly, view it on GitHub <#2 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BJ4CMV2MVLMKQKKZHF3SU332AIWXZAVCNFSM6AAAAABRS5ZOVWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINZRGA2TONZYHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[thmshmm]

Sry for the late reply. Did not test it but you could try something like this:

processors:
  transform:
    metric_statements:
      - context: metric
        statements:
          - set(start_time_unix_nano, time_unix_nano)

This should copy the time_unix_nano to start_time_unix_nano.