[log-ingestor] Improve ingestion job design #1733
Description
Request
As introduced in #1704, we need to pass a callback to create and spawn an ingestion job to adapt different type of job creation. This design complicates things. We should avoid using the callback in a future refactoring.
Possible implementation
Instead of creating and spawning ingestion jobs in the same method, we could split the creation and spawn so that:
- Ingestion jobs can be created in advance.
- We spawn the job after all validations passed.
In this way, no callback will be needed.
Splitting the spawn as a separate function may also help us handling failures: we don't need to create a new job instance but just need to re-spawn after log-ingestor detects and retries from a failure.