Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] Add ecs mapping for k8s.deployment.name #33622

Merged

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Jun 18, 2024

Description:

This PR adds a mapping for the k8s.deployment.name attribute which is enabled by default by the k8sattributesprocessor.
The idea is to have all the enabled by default attributes mapped to ECS properly, for consistency.

Link to tracking Issue: ~

Testing:

Tested locally with:

mode: daemonset
presets:
  logsCollection:
    enabled: true
    storeCheckpoints: true
  kubernetesAttributes:
    enabled: true

extraEnvs:
  - name: ES_ENDPOINT
    valueFrom:
      secretKeyRef:
        name: elastic-secret
        key: es_endpoint
  - name: ES_API_KEY
    valueFrom:
      secretKeyRef:
        name: elastic-secret
        key: es_api_key

image:
  repository: otelcontribcol-dev
  tag: "0.0.1"
  pullPolicy: IfNotPresent

command:
  name: otelcontribcol

config:
  exporters:
    debug:
      verbosity: detailed
    elasticsearch/logs:
      endpoints:
        - ${env:ES_ENDPOINT}
      api_key: ${env:ES_API_KEY}
      logs_index: otel_logs_index
      mapping:
        mode: ecs
      sending_queue:
        enabled: true
        num_consumers: 20
        queue_size: 1000
  receivers:
    filelog:
      start_at: end
      include_file_name: false
      include_file_path: true
      exclude:
        - /var/log/pods/default_daemonset-opentelemetry-collector*_*/opentelemetry-collector/*.log
      include:
        - /var/log/pods/*/*/*.log
      operators:
        - id: container-parser
          type: container
          output: some
  processors:
    k8sattributes:
      filter:
        node_from_env_var: K8S_NODE_NAME
      passthrough: false
      pod_association:
        - sources:
            - from: resource_attribute
              name: k8s.pod.ip
        - sources:
            - from: resource_attribute
              name: k8s.pod.uid
        - sources:
            - from: connection
      extract:
        metadata:
          - "k8s.namespace.name"
          - "k8s.deployment.name"
          - "k8s.node.name"
          - "k8s.pod.name"
          - "k8s.pod.uid"
  service:
    pipelines:
      logs:
        receivers: [filelog]
        processors: [batch]
        exporters: [elasticsearch/logs]

The documents are stored in ES properly:

{
    "@timestamp": "2024-06-18T10:57:45.307790375Z",
    "agent": {
      "name": "otlp"
    },
    "k8s": {
      "container": {
        "name": "busybox",
        "restart_count": "0"
      },
    },
    "kubernetes": {
      "deployment": {
        "name": "deploy-logs"
      },
      "namespace": {
        "name": "default"
      },
      "node": {
        "name": "kind-control-plane"
      },
      "pod": {
        "name": "deploy-logs-595cdc6b8b-dptrr",
        "uid": "5166a2e2-9152-4d86-b8a4-668cc2f49a39"
      }
    },
    "log": {
      "file": {
        "path": "/var/log/pods/default_deploy-logs-595cdc6b8b-dptrr_5166a2e2-9152-4d86-b8a4-668cc2f49a39/busybox/0.log"
      },
      "iostream": "stdout"
    },
    "logtag": "F",
    "message": "otel logs at 10:57:45",
    "time": "2024-06-18T10:57:45.307790375Z"
}

Documentation: ~

/cc @carsonip @lahsivjar @andrzej-stencel

@ChrsMark ChrsMark force-pushed the add_es_exporter_deploy_name_ecs branch from 721c9e6 to 4b5becf Compare June 18, 2024 11:40
@ChrsMark ChrsMark marked this pull request as ready for review June 18, 2024 11:44
@ChrsMark ChrsMark requested review from a team and Aneurysm9 June 18, 2024 11:44
@ChrsMark ChrsMark force-pushed the add_es_exporter_deploy_name_ecs branch 2 times, most recently from 363e311 to ff26bb6 Compare June 18, 2024 11:58
Copy link
Member

@lahsivjar lahsivjar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

andrzej-stencel pushed a commit that referenced this pull request Jun 21, 2024
…o ECS (#33670)

**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
This PR adds support for preserving resource attributes that are valid
ECS fields in addition of mapping them. At the moment the `host.name` is
mapped to the `host.hostname`. Both are valid ECS fields but can differ
in some cases. Since there is no such distinction in SemConv right now,
it does makes sense to preserve both.

refs:
-
https://www.elastic.co/guide/en/ecs/current/ecs-host.html#field-host-name
-
https://www.elastic.co/guide/en/ecs/current/ecs-host.html#field-host-hostname

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>

Using the testing notes from
#33622.

Stored document:
```json
 {
    "app": {
      "label": {
        "component": "migration-logger"
      }
    },
    "kubernetes": {
      "node": {
        "name": "kind-control-plane"
      },
      "pod": {
        "uid": "0eda57cd-a4ae-4e89-88fa-c771d3bf0c77",
        "name": "daemonset-logs-4sqjq"
      },
      "namespace": "default"
    },
    "agent": {
      "name": "otlp"
    },
    "@timestamp": "2024-06-20T07:27:42.589678923Z",
    "log": {
      "iostream": "stdout",
      "file": {
        "path": "/var/log/pods/default_daemonset-logs-4sqjq_0eda57cd-a4ae-4e89-88fa-c771d3bf0c77/busybox/0.log"
      }
    },
    "service": {
      "name": "migration-logger"
    },
    "k8s": {
      "container": {
        "restart_count": "0",
        "name": "busybox"
      },
      "pod": {
        "start_time": "2024-06-20T07:27:21Z"
      },
      "daemonset": {
        "name": "daemonset-logs"
      }
    },
    "host": {
      "hostname": "daemonset-opentelemetry-collector-agent-l6pzp",
      "os": {
        "type": "linux",
        "platform": "linux"
      },
      "name": "daemonset-opentelemetry-collector-agent-l6pzp"
    },
    "time": "2024-06-20T07:27:42.589678923Z",
    "message": "otel logs at 07:27:42",
    "logtag": "F"
  }
```

**Documentation:** <Describe the documentation added.> ~

/cc @lahsivjar @andrzej-stencel @carsonip

---------

Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
ChrsMark added 2 commits June 21, 2024 16:04
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
Signed-off-by: ChrsMark <chrismarkou92@gmail.com>
@ChrsMark ChrsMark force-pushed the add_es_exporter_deploy_name_ecs branch from 53f6c1e to 0b0b709 Compare June 21, 2024 13:07
@andrzej-stencel andrzej-stencel merged commit cf1c93e into open-telemetry:main Jun 21, 2024
154 checks passed
@github-actions github-actions bot added this to the next release milestone Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants