Need help on mutliline logs.

[karthikpb88]

What is a problem?

The application host is in pdt timezone when log pushed to OCI logging analytics (works only UTC timezone), we are facing issues with timestamp and bit confused about multiline formats and so on.

Describe the configuration of Fluentd

<source>
    @type tail
    @id in_xxxx
    path /var/log/xxxx.log
    pos_file /var/lib/fluent_oci_outplugin/pos/xxxxx.pos
    read_from_head false
    enable_watch_timer true
    enable_stat_watcher false
    tag oci.0.xxxx.log.*
    <parse>
        @type multiline
        format_firstline /^\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}\s+\[[^\]]+\]/
        format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<message>.*)$/
	time_key time
        time_format %Y-%m-%d %H:%M:%S
        keep_time_key true
        timezone -07:00
    </parse>
</source>

<filter oci.0.xxxxx.**>
    @type record_transformer
    enable_ruby true
    <record>
        oci_la_metadata ${{'Host Name (Client)':"#{hostname || 'UNKNOWN'}", 'Host IP Address (Client)':"xxxx", 'Environment':"xxxxxx"}}
        oci_la_log_group_id "xxxx"
        oci_la_log_path ${tag}
        oci_la_log_source_name "xxxxxx"
	message ${time} ${record["message"]}
    </record>
</filter>


<match oci.0.xxxxx.**>
    @type copy
    <store>
        @type oci-logging-analytics
        namespace xxxxx
        config_file_location /var/lib/td-agent/.oci/config
        profile_name DEFAULT
        plugin_log_location /var/log/fluent_oci_outplugin/logs/
        plugin_log_level info
        plugin_log_file_size 1MB
        plugin_log_file_count 10
        <buffer>
            @type file
            path /var/log/fluent_oci_outplugin/buffer/0/xxxxxx/
            disable_chunk_backup true
            flush_thread_count 4
        </buffer>
    </store>
    <store>
        @type flowcounter
        @id flowcounter_agg_0_xxxxxx
        @label @counts_0
        tag worker.0
        count_keys message
        aggregate all
        unit minute
    </store>
</match>

Describe the logs of Fluentd

Application log format:

2024-08-29 10:03:39 [Thread] Severity message
2024-08-29 09:48:31 [Thread] Severity message
[Start Packet Body]
XXX
XXX
XXX
[End Packet Body]

Error on td-agent logs:
2024-09-03 10:18:27 -0700 [warn]: #0 [xxx] pattern not matched: "2024-09-03 10:18:26 [thread | producer-1] DEBUG xxxx, clusterId='xxx', controllerId=1, topics=[xxxx', topicId=AAAAAAAAAAAAAAAAAAAAAA, isInternal=false, partitions=[MetadataResponsePartition(errorCode=0, partitionIndex=0, leaderId=1, leaderEpoch=163, replicaNodes=[1, 2, 3], isrNodes=[2, 1, 3], offlineReplicas=[]), MetadataResponsePartition(errorCode=0, partitionIndex=2, leaderId=3, leaderEpoch=141, replicaNodes=[3, 2, 1], isrNodes=[1, 3, 2], offlineReplicas=[]), MetadataResponsePartition(errorCode=0, partitionIndex=1, leaderId=2, leaderEpoch=120, replicaNodes=[2, 3, 1], isrNodes=[1, 3, 2], offlineReplicas=[])], topicAuthorizedOperations=-xxxxx)], clusterAuthorizedOperations=-xxxxxx)"

Environment

- Fluentd version:
- TD Agent version:
- Fluent Package version:
- Docker image (tag):
- Operating system:
- Kernel version:

[karthikpb88]

td and fluentd version:
td-agent 4.5.2 fluentd 1.16.3 (d3cf2e0)
OS:
NAME="Oracle Linux Server"
VERSION="8.10"
Kernal version:
Linux xxxxx 5.4.17-2136.332.5.2.el8uek.x86_64 #3 xxx Tue Jun 11 17:58:26 PDT 2024 x86_64 x86_64 x86_64 GNU/Linux
Docker image: N/A