How to write logs via File Exporter in non OTLP JSON format

[genadipost]

Component(s)

exporter/file

Describe the issue you're reporting

I would like to export logs to a file using OpenTelemetry Collector File Exporter , also i would like to format the logs in logfmt format.

I was able to export the logs to a file but i am unable to find documentation which explains how to change the format of the logs.

Logs (also traces and metrics) seems can be exported in OTLP JSON format only.

The data must be encoded according to the format specified in the OTLP JSON Encoding.
Only top-level objects, TracesData, MetricsData, and LogsData are supported.
Files must contain exactly one type of data: traces, metrics, or logs.
...
This is an example showing logs:
{"resourceLogs":[{"resource":{"attributes":[{"key":"resource-attr","value":{"stringValue":"resource-attr-val-1"}}]},"scopeLogs":[{"scope":{},"logRecords":[{"timeUnixNano":"1581452773000000789","severityNumber":9,"severityText":"Info","body":{"stringValue":"This is a log message"},"attributes":[{"key":"app","value":{"stringValue":"server"}},{"key":"instance_num","value":{"intValue":"1"}}],"droppedAttributesCount":1,"traceId":"08040201000000000000000000000000","spanId":"0102040800000000"},{"timeUnixNano":"1581452773000000789","severityNumber":9,"severityText":"Info","body":{"stringValue":"something happened"},"attributes":[{"key":"customer","value":{"stringValue":"acme"}},{"key":"env","value":{"stringValue":"dev"}}],"droppedAttributesCount":1,"traceId":"","spanId":""}]}]}]}

[github-actions]

Pinging code owners:

• exporter/file: @atingchen

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[VihasMakwana]

@genadipost you can utilise the textencoding extension.
I can help you with the config. Can you send me your current configuration?

[genadipost]

Hi @VihasMakwana, I am using opentelemetry-kube-stack to gather k8s logs and metrics.

This is the relevant part of the configuration:

    exporters:
        file:
            path: /otel-data/otel-exporter.log
            rotation:
                max_backups: 15
                max_megabytes: 50
    extensions:
        file_storage:
            directory: /var/lib/otelcol
    processors:
        batch:
            send_batch_max_size: 1500
            send_batch_size: 1000
            timeout: 1s
        k8sattributes:
            extract:
                labels:
                    - from: pod
                      key: app.kubernetes.io/name
                      tag_name: service.name
                    - from: pod
                      key: k8s-app
                      tag_name: service.name
                    - from: pod
                      key: app.kubernetes.io/instance
                      tag_name: k8s.app.instance
                    - from: pod
                      key: app.kubernetes.io/version
                      tag_name: service.version
                    - from: pod
                      key: app.kubernetes.io/component
                      tag_name: k8s.app.component
                metadata:
                    - k8s.namespace.name
                    - k8s.pod.name
                    - k8s.pod.uid
                    - k8s.node.name
                    - k8s.pod.start_time
                    - k8s.deployment.name
                    - k8s.replicaset.name
                    - k8s.replicaset.uid
                    - k8s.daemonset.name
                    - k8s.daemonset.uid
                    - k8s.job.name
                    - k8s.job.uid
                    - k8s.container.name
                    - k8s.cronjob.name
                    - k8s.statefulset.name
                    - k8s.statefulset.uid
                    - container.image.tag
                    - container.image.name
                    - k8s.cluster.uid
            filter:
                node_from_env_var: K8S_NODE_NAME
            pod_association:
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.uid
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.name
                    - from: resource_attribute
                      name: k8s.namespace.name
                    - from: resource_attribute
                      name: k8s.node.name
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.ip
                - sources:
                    - from: resource_attribute
                      name: k8s.pod.name
                    - from: resource_attribute
                      name: k8s.namespace.name
                - sources:
                    - from: connection
        resourcedetection/env:
            detectors:
                - env
            override: false
            timeout: 2s
    receivers:
        filelog:
            exclude: []
            include:
                - /var/log/pods/*/*/*.log
            include_file_name: false
            include_file_path: true
            operators:
                - id: container-parser
                  max_log_size: 102400
                  type: container
            retry_on_failure:
                enabled: true
            start_at: end
            storage: file_storage
    service:
        extensions:
            - file_storage
        pipelines:
            logs:
                exporters:
                    - file
                    - debug
                processors:
                    - k8sattributes
                    - resourcedetection/env
                    - batch
                receivers:
                    - filelog

[VihasMakwana]

@genadipost can you try this out? Haven't tested it, but it should work.

exporters:
    file:
        encoding: text_encoding
        path: /otel-data/otel-exporter.log
        rotation:
            max_backups: 15
            max_megabytes: 50
extensions:
    text_encoding:
        encoding: utf8

    file_storage:
        directory: /var/lib/otelcol
processors:
    batch:
        send_batch_max_size: 1500
        send_batch_size: 1000
        timeout: 1s
    k8sattributes:
        extract:
            labels:
                - from: pod
                    key: app.kubernetes.io/name
                    tag_name: service.name
                - from: pod
                    key: k8s-app
                    tag_name: service.name
                - from: pod
                    key: app.kubernetes.io/instance
                    tag_name: k8s.app.instance
                - from: pod
                    key: app.kubernetes.io/version
                    tag_name: service.version
                - from: pod
                    key: app.kubernetes.io/component
                    tag_name: k8s.app.component
            metadata:
                - k8s.namespace.name
                - k8s.pod.name
                - k8s.pod.uid
                - k8s.node.name
                - k8s.pod.start_time
                - k8s.deployment.name
                - k8s.replicaset.name
                - k8s.replicaset.uid
                - k8s.daemonset.name
                - k8s.daemonset.uid
                - k8s.job.name
                - k8s.job.uid
                - k8s.container.name
                - k8s.cronjob.name
                - k8s.statefulset.name
                - k8s.statefulset.uid
                - container.image.tag
                - container.image.name
                - k8s.cluster.uid
        filter:
            node_from_env_var: K8S_NODE_NAME
        pod_association:
            - sources:
                - from: resource_attribute
                    name: k8s.pod.uid
            - sources:
                - from: resource_attribute
                    name: k8s.pod.name
                - from: resource_attribute
                    name: k8s.namespace.name
                - from: resource_attribute
                    name: k8s.node.name
            - sources:
                - from: resource_attribute
                    name: k8s.pod.ip
            - sources:
                - from: resource_attribute
                    name: k8s.pod.name
                - from: resource_attribute
                    name: k8s.namespace.name
            - sources:
                - from: connection
    resourcedetection/env:
        detectors:
            - env
        override: false
        timeout: 2s
receivers:
    filelog:
        exclude: []
        include:
            - /var/log/pods/*/*/*.log
        include_file_name: false
        include_file_path: true
        operators:
            - id: container-parser
                max_log_size: 102400
                type: container
        retry_on_failure:
            enabled: true
        start_at: end
        storage: file_storage
service:
    extensions:
        - file_storage
        - text_encoding
    pipelines:
        logs:
            exporters:
                - file
                - debug
            processors:
                - k8sattributes
                - resourcedetection/env
                - batch
            receivers:
                - filelog

[genadipost]

@VihasMakwana It seems that text_encoding is not part of any distribution.

https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-contrib/manifest.yaml

https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol-k8s/manifest.yaml

https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-otlp

https://github.com/open-telemetry/opentelemetry-collector-releases/blob/main/distributions/otelcol/manifest.yaml

[VihasMakwana]

Oh!

I'll have a chat with codeowners and discuss this.

[genadipost]

@VihasMakwana Any ticket to follow on?