Allow for http server to publish events and http client to subscribe to events

[Lance-Drane]

Terminology

• "broker2http" is the current name of the HTTP server
• "http2broker" is the current name of the HTTP client

Currently

We currently have the following setup:

1. broker2http subscribes to Message Broker 1, and creates a Server-Sent Event endpoint which will emit messages originating from the same Service as the broker2http configuration. The SSE endpoint requires authentication to subscribe to.
2. http2broker subscribes to the SSE endpoint created in 1) , and when it gets another message, publishes it to Message Broker 2

Problem

This setup is fairly flexible and can meet many usecases, but has a couple of problems:

• You have to deploy four proxy applications (two on each server), assuming you want a two-way data flow (if one way, you can remove one "pair"):

|-------------|   <- broker2http(1)      |       <- http2broker(1)  ->  |-------------| 
| BROKER|                 (system 1)  |  (system2)                       | BROKER|
|-------------|   <- http2broker(2) ->  |      broker2http(2)       ->  |-------------| 

• Assuming you want a two-way data flow, this solution requires for both system1 and system2 to allow egress from each http2broker (usually not too hard to do), and to each allow ingress from each broker2http (much harder, and requires specialized firewall rules)

Solution

We need to allow the following additional steps:

1. broker2http creates an HTTP POST endpoint which can publish a message on Message Broker 1. This endpoint requires authentication to use. NOTE: if using RabbitMQ, you can potentially use the HTTP publish approach if you are fine with sharing broker credentials across services.
2. http2broker subscribes to Message Broker 2, and when it gets a message from the same Service as its own configuration, sends POST requests to the endpoint mentioned in 1)

This approach solves a few useful problems:

• You only need to deploy a single proxy application, assuming you only want two brokers to communicate with one another. (Note that you will need to deploy all four applications if seeking a truly dynamic setup where any arbitrary system can talk to any other arbitrary system, but for purposes of having two specific systems communicate with one another, this is fine.)
• More importantly, if one of the services does not have strict ingress rules, you can deploy the web server on the system with lenient ingress and deploy the HTTP client on the system with strict ingress. This allows you to quickly work around firewall rules.

---

ping @marshallmcdonnell