[Q&A] read_from_head ignored using tail plugin???

[rickFanta]

What is a problem?

Using tail plugin, and polling (i.e. not inotify yet). There is no log file rotation. I realize that multiple workers can be used to break up this path further (e.g. even/odd, every third file, etc) to limit scope of the search for new files and updates.

What we're seeing though in 12-50% of cases (depending on load) that fluentD misses the beginning part of the file and only after seeing a log message like the below are the lines seen and sent by fluentD.

e.g. if we see this:

td-agent.log:2024-08-19 13:39:10 -0400 [info]: #0 following tail of /my/long/file/path/with/ahundred_thousand_subdirs/job-NNNNN-task-1.out

... then all lines from the start of the log (e.g. 13:34:00-0100 and onwards until the above time get missed.

I understand the latency can be large in polling such a large hierarchy, but would expect the logs to be complete once noticed by FluentD if read_from_head true is set.

I've looked through the Coolio and FileWatcher code in FluentD but do not see an obvious issue.

Describe the configuration of Fluentd

FLuentD Ruby - 1.16.1

<source>
        @type tail
        path  /my/long/file/path/with/fifty_thousand_subdirsA/log/job-*/task*/*.out,/my/long/file/path/with/fifty_thousand_subdirsB/log/job-*/task*/*.out
        tag topic-blah-bleh
        path_key tailed_path
        pos_file /my/pos_file_path/td-agent/positionfile/blah.bleh.raw.pos
        pos_file_compaction_interval 24h
        read_from_head true
        follow_inodes true
        format none
        limit_recently_modified 4d
        @label my_label
</source>

Describe the logs of Fluentd

td-agent.log:2024-08-19 13:39:10 -0400 [info]: #0 following tail of /my/long/file/path/with/ahundred_thousand_subdirs/job-NNNNN-task-1.out

Environment

- Fluentd version: 1.16.1
- TD Agent version:
- Fluent Package version: 1.16.1
- Docker image (tag):
- Operating system: RHEL-7
- Kernel version:  Linux version 3.10.0-1160.108.1.el7.x86_64