What is Kiwix/openZIM standard message broker?

[benoit74]

This is kinda an old topic, but it looks like it might now be time to make a decision, create a plan and start implementing it.

This issue arises now because @elfkuzco came to the conclusion that it begins to be too complex to handle messaging between CMS components through DB and proposed to introduce a real messaging solution in openzim/cms#201

This is an old topic already tracked in openzim/zimfarm#1116 for the Zimfarm, and identified like "years ago".

What I've already identified in terms of requirements:

• throughput is expected to be very moderate, in the order of magnitude of few message per seconds
• we need persistence + backup
• we need at least once delivery guarantee ; up to the producer / consumer to achieve exactly once delivery if needed, typically with idempotent code
• we need a solution with acknowledgement ; consumer acknowledge the message has been received before the broker consider it as delivered
• we want this solution to be used only when the message consumer is a technical component and does not already exposes an API ; it will not replace traditional API communication
• we need some form of security to prevent unauthorized access to the messages

What we probably don't need:

• HA solution (like clustering, ...)
• a solution capable to cope with very big messages (we can ask apps to generate messages in the range of 1K or less, or rely on another protocol)
• a solution accessible by web interfaces (end-user UIs)

I assume we have some requirements linked to our k8s infra but it is very unclear to me. Do we have to require HTTPS (+Websocket) or can we accept any protocol (meaning we will expose it directly on the host port, like we already do for other services)?

In terms of plan, I propose that while we identify a good candidate solution, implement it in the CMS first and from there decide if we continue with this solution or change to another alternative. It could be a good moment to also decide if we deploy a central component or continue with one component per app.

@elfkuzco @rgaudin @kelson42 can you help me better identify requirements?

[kelson42]

I support the idea of a central infrastructure component based on an "industry standard" FOSS piece of software. For the rest I don't believe to have much more to say on this (at this stage). @rgaudin @benoit74 Once you kind of agree on a good (short list of) candidate(s), please report here for final discussion.

[rgaudin]

I don't think we should try to find a standard Kiwix way at this stage.