Preventing Duplicate Logs in OpenSearch When Tailing the Same Log File from Multiple Fluentd Servers over NFS

[ramazancetinn]

What is a problem?

Hello,

I'm currently working on setting up log collection and indexing for my application, and I've run into a challenge that I'm hoping to get some advice on.

Environment Setup:

I have three Fluentd servers running in parallel.
All three servers have a mount point to the same NFS server.
On this NFS mount, there's a log file (app.log) generated by our application.
I want to use Fluentd to tail this log file and push the logs to OpenSearch for indexing and analysis.

The Problem:

When I configure each Fluentd server to tail the log file from the NFS mount and use a shared pos_file (position file) also on the NFS mount, I'm encountering an issue where each log entry is sent to OpenSearch three times—once from each Fluentd server. This results in duplicate entries in OpenSearch, which is not ideal.

What I've Observed:

• Each Fluentd server reads the same log entries and sends them independently to OpenSearch.
• Sharing the pos_file doesn't prevent duplicates and might be causing race conditions.
• Disabling the tail input on two servers stops the duplication but introduces a single point of failure.

What I'm Trying to Achieve:

• Eliminate duplicate log entries in OpenSearch.
• Maintain high availability so that if one Fluentd server fails, the others can continue processing the logs.
• Use the existing infrastructure without introducing significant new components (e.g., message queues like Kafka).

Questions for the Community:

• How can I configure Fluentd to prevent duplicate logs when multiple instances are tailing the same file over NFS?
• Is there a way to coordinate the Fluentd instances so that only one tails the file at a time, but failover to others if the primary fails?
• Should I avoid sharing the pos_file among the Fluentd instances? If so, what's the best practice for position file management in this scenario?
• Are there alternative Fluentd plugins or input methods that would better suit this use case?

Considerations:

• I've read that sharing pos_file between multiple instances can cause issues due to file locking and race conditions.
• I'm looking for a solution that minimizes changes to the current setup and keeps things relatively simple.
• High availability is important, but data integrity (no duplicates) is critical.

Possible Solutions I'm Considering:

• Designate a Primary Fluentd Instance: Configure only one Fluentd server to tail the log file, but I'm concerned about the lack of redundancy.
• Use a Coordination Service: Implement leader election using tools like ZooKeeper or Consul, but this adds complexity.
• External Deduplication: Allow duplicates but use OpenSearch features to filter them out, though this might impact performance.

Any advice, recommendations, or insights would be greatly appreciated!

Thank you in advance for your help!

Describe the configuration of Fluentd

Current Fluentd Configuration:

Here's a simplified version of my Fluentd configuration on each server:

<source>
  @type tail
  path /mnt/nfs/app.log
  pos_file /mnt/nfs/fluentd.pos
  tag app.logs
  read_from_head true
  <parse>
    @type json
  </parse>
</source>

<match app.logs>
  @type opensearch
  host opensearch.example.com
  port 9200
  scheme https
  index_name app-logs-%Y%m%d
  type_name _doc
  logstash_format true
  include_timestamp true
</match>

Describe the logs of Fluentd

No response

Environment

- Fluentd version: fluentd 1.16.3
- Fluent Package version: fluent-package 5.0.2
- Operating system:
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.3 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal
- Kernel version: 5.4.0-173-generic

[daipom]

pos_file should not be shared.

Is it for fault tolerance to use multiple instances to tail the same file?

[ramazancetinn]

Yes, we’re using multiple instances to tail the same file for fault tolerance. We understand that sharing the pos_file is not recommended, but when we use separate pos_file files for each instance, we encounter duplicate logs in OpenSearch. Our goal is to prevent duplication while maintaining high availability

[daipom]

I see.
It is not assumed to run multiple Fluentd for fault tolerance.

The single instance has some degree of fault tolerance.

• When a worker process dies, the supervisor process restarts it immediately.
• Then, in_tail restarts tailing from the previous position. So, it can collect all logs correctly after all.

[ramazancetinn]

Thank you for your reply.