GSoC'22: Log File Rotation

[shikharvashistha]

Description

This pull request is intended to introduce a log file rotation mechanism for the syslog-ng application which will solve the current issue of log files becoming very large and thus affecting the performance of the application.

Overview of changes

The following changes have been made in order to implement the solution:

• LogProtoFileWriter class have been modified to connect to the existing SignalSlotConnector class in order to receive notifications via log_proto_file_writer_new() function when a log file is opened and will carry filename and reopener pointer as user_data.

• A new slot have been added to the SignalSlotConnector class which will receive notifications when a log file is opened and will call a new function log_proto_file_writer_reopen() in order to check if the opened file needs to be rotated and all the necessary logic is applied to rotate the log file for which the file-rotation functionality was built.

Configuration snippets

Whole configuration file

#############################################################################
# Default syslog-ng.conf file which collects all local logs into a
# single file called /var/log/messages.
#

@version: 3.37
@include "scl.conf"

destination d_file {
    file("/tmp/temp.log", file-rotation(size(0KB), interval("daily"), dateformat("-%Y-%m-%d"), rotate(2)));
};

source my_tcp {
    stdin();
};


log {
source(my_tcp);
destination(d_file);
};

Necessary changes

destination d_file {
    file("/tmp/temp.log", file-rotation(size(0KB), interval("daily"), dateformat("-%Y-%m-%d"), rotate(2)));
};

Alternatives/Problem faced

1. lseek vs fstat call. Can refer the discussion here

2. Attaching a reopner function to the LogProtoFileWriter class vs using a global flag to signal file-rotation plugin to rotate the file.

Discussions

• Pull Request - 1

• Pull Request - 2

Resolves #2964

[kira-syslogng]

This user does not have permission to start the build. Can one of the admins verify this patch and start the build?
(admin: you have the next options (make sure you checked the code):
"ok to test" to accept this pull request (and further changes) for testing
"test this please" for a one time test run
"add to whitelist" add author of a Pull Request to whitelist (globally, be careful, it means this user can trigger kira for any PR)
do nothing -> CI won't start)

[kira-syslogng]

This user does not have permission to start the build. Can one of the admins verify this patch and start the build?
(admin: you have the next options (make sure you checked the code):
"ok to test" to accept this pull request (and further changes) for testing
"test this please" for a one time test run
"add to whitelist" add author of a Pull Request to whitelist (globally, be careful, it means this user can trigger kira for any PR)
do nothing -> CI won't start)

[github-actions]

No news file has been detected. Please write one, if applicable.

[MrAnno]

@kira-syslogng ok to test

[MrAnno]

@kira-syslogng do stresstest

[kira-syslogng]

Kira-stress-test: Build SUCCESS

[bazsi]

I think the reopener should probably be a pointer here. Also I'd look into encapsulating the last three arguments into
a struct of some kind, which all relates to rotation support.

[shikharvashistha]

Okay sure

[bazsi]

the slot is probably better placed in the FileOpener constructor so that if an implementation of construct_dst_proto needs it it can find it in self and we wouldn't need to change the prototype here.

I guess this only applies to regular files and not so much to /proc/kmsg or named pipes.

[bazsi]

I think we could allow the file opener plugin to indicate that a reopen is needed via a return value, so we wouldn't have to push the FileReopener instance down to the guts of the rotation plugin.

This way the reopen would either be delegated to the LogProtoFileWriter or even better back to the driver via the use of the log_pipe_notify() mechanism.

[shikharvashistha]

Do you mean that we can have a returning flag of some sort for the purpose of reopening the file ?

[bazsi]

Those are generated by bison during compilation. The easiest way to generate a source tar.gz that has them is via dbld. in the source directory: `./dbld/rules tarball` You can also launch a "dev" docker environment by issuing: `./dbld/rules shell` And then to bootstrap compilation, within the shell: `$ /dbld/bootstrap` Once that finishes you can compile syslog-ng along with all necessities generated using the make command in the /build directory. HTH, Bazsi

…

On Wed, Oct 18, 2023 at 4:55 AM shalgf ***@***.***> wrote: Dear shikharvashistha: Thank you for your feature Log File Rotation, it is very helpful for me. I get your submittion and compile, found that there is not file-rotation in Makefile.in, and there is not files file-rotation-grammar.y, file-rotation-grammar.c and file-rotation-grammar.h, Will you please tell me how can I generate them? Thanks shalgf — Reply to this email directly, view it on GitHub <#4145 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFOK5XBUEGQAC3UB7C2RFLX75AKFAVCNFSM6AAAAAAQRB3FROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRXGUZTOMBXGI> . You are receiving this because your review was requested.Message ID: ***@***.***>

[tuxillo]

What happened to this ?

[mehul-m-prajapati]

@HofiOne @bazsi : I can work on the remaining part of this feature. Do you want to have this log rotation ?

[czanik]

Thanks @mehul-m-prajapati for your offer. However, we discussed this question internally, and we do not want to continue this effort. If we ever get to this, it will be a different implementation.