You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello there!
I use fluent-plugin-s3 and according my configuration it should send logs from few files to MinIO once per day.
Hers is the configuration:
<match projectname>
@type s3
aws_key_id "#{ENV['MINIO_ROOT_USER']}"
aws_sec_key "#{ENV['MINIO_ROOT_PASSWORD']}"
s3_bucket tenants
format json
force_path_style true
s3_endpoint "http://#{ENV['MINIO_HOST']}:#{ENV['MINIO_PORT']}/"
path "#{ENV['TENANT_ID']}/logs/projectname-" # This prefix is added to each file
time_slice_format %Y%m%d%H%M # This timestamp is added to each file name
#s3_object_key_format %{path}%{time_slice}.%{file_extension} # Should be commented because target log file will be overwritten few times and logs wil be lost.
<buffer tag,time>
@type file
path /fluentd/logs/
timekey 1440m
timekey_wait 10m
flush_mode lazy
timekey_use_utc true
chunk_limit_size 256m
</buffer>
</match>
I noticed that fluent-plugin-s3 often creates on MinIO more than one file per day.
Example,
projectname-202401240532_3.gz
projectname-202401240532_2.gz
projectname-202401240532_1.gz
projectname-202401240532_0.gz
I would like to have single log file on MinIO per day.
To achieve this goal I tried to play with s3_object_key_format property.
default value = %{path}%{time_slice}_%{index}.%{file_extension}
I changed it to %{path}%{time_slice}.%{file_extension}. As result I lost part of the logs. Looks like target log file was overwritten few times and I saw only data from the latest iteration.
How to force fluent-plugin-s3 to create only single file on MinIO when timekey condition has been met (and not lose data)?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
-
Hello there!
I use fluent-plugin-s3 and according my configuration it should send logs from few files to MinIO once per day.
Hers is the configuration:
I noticed that fluent-plugin-s3 often creates on MinIO more than one file per day.
Example,
projectname-202401240532_3.gz
projectname-202401240532_2.gz
projectname-202401240532_1.gz
projectname-202401240532_0.gz
I would like to have single log file on MinIO per day.
To achieve this goal I tried to play with s3_object_key_format property.
default value = %{path}%{time_slice}_%{index}.%{file_extension}
I changed it to %{path}%{time_slice}.%{file_extension}. As result I lost part of the logs. Looks like target log file was overwritten few times and I saw only data from the latest iteration.
How to force fluent-plugin-s3 to create only single file on MinIO when timekey condition has been met (and not lose data)?
Beta Was this translation helpful? Give feedback.
All reactions