Endpoints may miss events during initial config sync/replay logs

[yhabteab]

Describe the bug

When the master has to replay a huge number of logs, which can take up to two to three minutes or even more, the remote peer will never receive live generated events from the local endpoint and may lose important events.

ApiListener::RelayMessageOne() determines, as stated in its doc block, whether a message has been successfully relayed or not. This allows the caller to decide whether the message should be saved to disk or not (replay log).

icinga2/lib/remote/apilistener.cpp

Lines 1211 to 1214 in 9e31b8b

	* @return true if the message has been relayed to all relevant endpoints,
	* false if it hasn't and must be persisted in the replay log
	*/
	bool ApiListener::RelayMessageOne(const Zone::Ptr& targetZone, const MessageOrigin::Ptr& origin, const Dictionary::Ptr& message, const Endpoint::Ptr& currentZoneMaster)

Once it reaches this point, it is considered as being relayed yet before it calls ApiListener::SyncSendMessage(). Keep in mind, it hasn't been checked whether the target endpoint is in syncing state so far.

icinga2/lib/remote/apilistener.cpp

Lines 1306 to 1308 in 9e31b8b

	relayed = true;
	SyncSendMessage(targetEndpoint, message);

ApiListener::SyncSendMessage() ultimately checks whether the message should be forwarded or not, i.e. when the endpoint is in syncing state, it will simply ignore this message and never try it again. Unless there are unconnected endpoints, this message will not be cached and the endpoint will never get to see this event.

icinga2/lib/remote/apilistener.cpp

Lines 1179 to 1183 in 9e31b8b

	void ApiListener::SyncSendMessage(const Endpoint::Ptr& endpoint, const Dictionary::Ptr& message)
	{
	ObjectLock olock(endpoint);
	if (!endpoint->GetSyncing()) {