Add sink event buffer support

[bhavya-tech]

We are calling the dump_events for every celery event received.

This is necessary for SocketioEventSink where we need to push the events to clients immediately. However, for sinks like FileEventSink, this would cause unnecessary overhead of opening the file and appending to it. (The filesystem cache comes into play so we will not incur frequent disk writes; but the kernel mode switch for file operations will still be there.)

Solution:

• We will have a configurable event buffer which will cache the events and dump it periodically.
• We can have two variables event_buffer_size (number of events to cache) and sink_dump_frequency (time interval at which buffer must be flushed.)
  • The buffer will be flushed when either of the two conditions are met.

Points to consider:

• How will these variables be set?
• Should these config be common for all sinks or individual for each sink?
  • Most probably will be individual for each sink as some sinks need to send them immediately while others can have delay
  • In case of individual for each sink:
  • Should we have a dedicated buffer for each sink? (Easier implementation, but consumes more memory.)
  • Or should we have a common buffer? (Memory efficient, but we need to keep track of which events have been dumped by which sink. Maybe use refcounting or bitmask? And delete the events which have been dumped by all sinks.)