Kafka Executor
Responsible for implementing the "ProcessExecutor" interface and is capable of managing the shutdown behaviour of any local components. Although at minimum requires a "ResultsListener" to return result futures.
To "execute" a process it generates a PiInstance with a new ID locally, sends a ReduceRequest with that PiInstance then uses a promise listener to wait and respond with the result or exception.
Contains PiInstance which need reducing, along with any returned AtomicProcess results which need to be posted to that instance.
Consumes the ReduceRequest topic, posts the results into the PiInstances and reduces them. Produces Assignment messages for AtomicProcesses when opening new threads, otherwise posts to "Results" topic with results. It also posts the latest PiInstances to the "PiiHistory" topic.
A stream of AtomicProcess calls which need computing. Contains the process to call and the arguments to use. It also contains the information necessary to uniquely identify id: the PiInstance ID and the call ID (an incrementing id for each AtomicProcess call made by that PiInstance).
Consumes the Assignment topic, executes AtomicProcesses, and posts SequenceRequests.
Stores:
- PiiUpdate: The latest PiInstance state for a given PiInstance.
- SequenceRequest: All the results of AtomicProcess calls which need posting to its PiInstance.
- SequenceFailures: The outstanding tasks which need mopping up after an exception.
Consumes the PiiHistory topic, matches up any pending SequenceRequests with the latest PiInstance and posts ReduceRequests for the Reducer.
NOTE: U = PiiUpdate, SR = SequenceRequest, Red = For "Red" PiInstance, Blue = For "Blue" PiInstance
The challenge with the Sequencer is that if multiple PiInstances are assigned to a single PiiHistory partition the resultant ReduceRequests cannot always be sent immediately (see above). If a SequenceRequest arrives without a matching PiiUpdate, we must wait to commit until one is present or it could be forgotten in event of a shutdown. PiiUpdates may be sent as they will return after a redundant pass through the reducer.
Should really be called "PiiLogs" or "PiEvents", stores the history of all PiEvents .
"ResultsListener"s does no work per se, each one has it's own ConsumerGroup so each ResultsListener receives all PiEvents (a broadcast/subscribe pattern). Implements the "PiObservable" trait for the cluster and the KafkaExecutor.