prometheus and splunk intergration-

[Davinder2609]

TYPE confluent_kafka_server_request_bytes gauge

[Davinder2609]

used below configration
scrape_configs:

• job_name: Confluent Cloud
  scrape_interval: 1m
  scrape_timeout: 1m
  honor_timestamps: true
  static_configs:
  • targets:
    • api.telemetry.confluent.cloud
      scheme: https
      basic_auth:
      username:
      password:
      metrics_path: /v2/metrics/cloud/export
      params:
      "resource.kafka.id":
    • lkc-1
    • lkc-2

[Davinder2609]

In Graph I can see all mertics .
but localhost/9090 I am not able to see
confluent_kafka_server_received_bytes
confluent_kafka_server_sent_bytes
confluent_kafka_server_received_records
confluent_kafka_server_sent_records
confluent_kafka_server_retained_bytes
confluent_kafka_server_request_count
confluent_kafka_server_active_connection_count
confluent_kafka_server_partition_count

[Davinder2609]

@Dabz Please advise on this issue

[Dabz]

Hi @Davinder2609 , as mentioned in the README, this project is no longer maintained and I encourage switching to the Confluent Cloud export endpoint that expose OpenMetrics data.

I would be glad to help if required, but I struggle to understand the problem, if you are able to see those metrics in Prometheus, it indicates a successful fetch & scrape.

[Davinder2609]

Thanks @Dabz ,with ccloud_exporter we are able to get all cloud metrics in splunk.but with promethueus we are not getting all confluent cloud metrics in splunk,only scrape_duration_seconds,scrape_samples_post_metric_relabeling,scrape_samples_scraped,scrape_series_added listed in splunk.
below mertics not listed-
confluent_kafka_server_received_bytes
confluent_kafka_server_sent_bytes
confluent_kafka_server_received_records
confluent_kafka_server_sent_records
confluent_kafka_server_retained_bytes
confluent_kafka_server_request_count
confluent_kafka_server_active_connection_count
confluent_kafka_server_partition_count

[Dabz]

Check the ccloudexporter logs, but very likely something is failing.
The metrics you list are the one that are always returned, regardless of the success of the queries to the Metrics API.

[Davinder2609]

We are not using ccloudexporter,Only using
Receiver- promethues
Exporter- splunk_hec/metrics

[Davinder2609]

using these 2 files

otel-collector-config.yml

receivers:
prometheus_simple:
collection_interval: 60s
endpoint: prometheus:9090

   metrics_path: /federate

   params:
    match[]:
      - '{job="ccloud_metrics"}'
      - '{__name__=~"job:.*"}'

otlp:
    protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318

exporters:
splunk_hec/metrics:
# Splunk HTTP Event Collector token.
token: "************************"

    # URL to a Splunk instance to send data to.
    endpoint: "https://*****splunkcloud.com:8088/services/collector"
    source: "http:kafkademo"
    sourcetype: "manual"
    index: "main"
    max_connections: 20
    disable_compression: false
    timeout: 10s      
    insecure_skip_verify: true

processors:
batch:

extensions:
health_check:
endpoint: 0.0.0.0:13133
pprof:
endpoint: :1888
zpages:
endpoint: :55679

service:
extensions: [pprof, zpages, health_check]
pipelines:
metrics:
receivers: [prometheus_simple]
processors: [batch]
exporters: [splunk_hec/metrics]

Prometheus.yml

scrape_configs:

• job_name: Confluent Cloud
  scrape_interval: 60s
  honor_labels: true
  honor_timestamps: true

  static_configs:

  • targets:
    • api.telemetry.confluent.cloud

  scheme: https
  basic_auth:
  username: *********
  password: *************

  metrics_path: /v2/metrics/cloud/export

  params:
  resource.kafka.id:
  - **********
  resource.schema_registry.id:
  - **********

  tls_config:
  insecure_skip_verify: true

[Davinder2609]

@Dabz Please advise!