Add support for cancelling compression jobs.

[junhaoliao]

Request

Add support for cancelling compression jobs. Currently, the compression scheduler does not handle job cancellation requests, meaning users cannot stop a running or pending compression job gracefully. This is essential for resource management, allowing users to terminate accidental or lower-priority jobs to free up resources for other tasks.

Possible implementation

1. Update Database Schemas/Constants:

   • Add CANCELLING and CANCELLED states to CompressionJobStatus in constants.py.
   • Add CANCELLED state to CompressionTaskStatus.

2. Update Task Managers:

   • Add a cancel() method to the abstract TaskManager interface.
   • Implement cancel() in CeleryTaskManager using celery_result.revoke(terminate=True).
   • Implement cancel() in SpiderTaskManager.

3. Update Compression Scheduler:

   • Implement a fetch_cancelling_jobs function to query jobs in the CANCELLING state.
   • Implement handle_cancelling_jobs logic in the main scheduler loop:
     • Identify active jobs that are marked as CANCELLING.
     • Call job.result_handle.cancel() to revoke dispatched tasks.
     • Update status of RUNNING and PENDING tasks to CANCELLED in the database.
     • Update the job status to CANCELLED and record the final duration.

[junhaoliao]

@sitaowang1998 could you comment on this? what needs to be done in spider to cancel a running task?

[sitaowang1998]

• Spider supports "cancelling" a job, which sets the job state to cancel, and no tasks will be scheduled for this job.
• Spider currently does not support preemptively terminating running tasks. It is planned but never implemented.
• The cancel is currently in C++ client interface only. It is easy to add the function in Python client interface.