Implement graceful shutdown for workflow engine

[khushboobhatia01]

I would like to suggest a new implementation plan for graceful shutdown of workflow engine.

Current implementation:

• On receiving SIGTERM workflow engine will exit abruptly.
• Any request that was in progress during shutdown will get stuck, causing the workflow execution to be in running state forever.

New implementation suggestion:

• On shutdown, the workflow engine should remove itself from the service registry.
• If there are other workflow engines in the service registry, then do the shutdown sequence. The shutdown sequence should be something like stop accepting incoming requests and wait until this workflow engine completed processing any requests (not some arbitrary sleep).
• Else if there are no other workflow engine in the service registry, then the workflow engine needs to stop accepting requests, pause all running workflow executions, and then proceed with rest of the shutdown sequence.
• On workflow engine startup, any workflow executions paused by the shutdown should be resumed.
  (Thanks @m4dcoder for the above flow suggestion Implement proper workflow engine shutdown #5367 (review))

Implementation plan:

1. Implement deregistration process for a service. This will be a pre-teardown job where on shutdown, service will deregister itself from the service registry.
2. Implementation of shutdown sequence. This will include checking of ongoing requests being handled by the workflow engine and then wait for these requests to complete and then terminate the process.
3. Identifying/Implementing a way to know if the workflow was paused by shutdown sequence or by user themselves. This will be required in case the last workflow engine in the service registry is shutting down and there are no more engines to serve requests.
4. Binding above 3 implementations together.

[m4dcoder]

@khushboobhatia01 Thank you for putting this together. The implementation plan looks right at high level. I suggest breaking this work into multiple smaller PRs. Here are a few additional comments.

• For 1, just to be on the same page, can we implement the deregistration for all the services and not just the workflow engine service?
• When the workflow engine is working on request for a workflow execution, it will be helpful if the workflow engine can update the hostname and process ID in the runner_info of the liveaction/action execution that corresponds to the workflow execution. The scheduler can periodically checks (and a CLI command that operators can run to check) the service registry and abandon orphaned workflow execution stuck in some running/canceling/pausing state.
• For 3, adding some context info in the action execution that the pause was initiated by system shutdown is easier then adding a new execution status enum because the latter will cascade into the state machine in orquesta and we want to minimize changes there when possible.

[khushboobhatia01]

Acknowledged @m4dcoder. Will start with the implementation work and share the PRs incrementally. Any further feedback is much appreciated.