Problem: when transfers are launched concurrently, bag validation fails randomly.

[scollazo]

Describe the bug

When capacity is set to a number bigger than one, and a bunch of transfers are loaded into minio, there is the following error for some of them in the ui:

System error: Error attempting to validate the Bag

And the following error in preprocessing temporal logs:

{
  "message": "bagvalidate: runner is busy",
  "source": "GoSDK",
  "stackTrace": "",
  "encodedAttributes": null,
  "cause": null,
  "applicationFailureInfo": {
    "type": "",
    "nonRetryable": false,
    "details": null
  }

To Reproduce

• Set the enduro capacity to a number bigger than 1 (I used 8), and restart enduro and the enduro-am-worker
• Launch 10-20 transfers (they are the same transfer, renamed):

for i in $(seq 1 10); do mc cp enduro/transfers/bag-7537ab2c-4e6b-4820-95bf-bd2c577351c3.zip enduro/transfers/concurrent$i.zip; done

Expected behavior

The transfers work

Screenshots

Additional context

Add any other context about the problem here.

[scollazo]

Log from preprocessing:

preprocessing-worker.log:Feb 12 10:01:33 sfa-dev preprocessing-worker[1146660]: {"level":"2","ts":1739354493.8953083,"logger":"preprocessing-worker.preprocessing-temporal","caller":"log/with_logger.go:84","msg":"error","Namespace":"default","TaskQueue":"preprocessing","WorkerID":"1146660@sfa-dev@","WorkflowType":"preprocessing","WorkflowID":"preprocessing-538c2ea6-57cd-47c8-95a8-2a7bd5e6d011","RunID":"cd385462-1793-49b8-a314-e89178ca9643","Attempt":1,"message":"activity error (type: bag-validate, scheduledEventID: 12, startedEventID: 13, identity: 1146660@sfa-dev@): bagvalidate: runner is busy","error":"System error","stacktrace":"go.temporal.io/sdk/log.(*withLogger).Error\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/log/with_logger.go:84\ngo.temporal.io/sdk/log.(*withLogger).Error\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/log/with_logger.go:84\ngo.temporal.io/sdk/internal/log.(*ReplayLogger).Error\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/log/replay_logger.go:79\ngithub.com/artefactual-sdps/preprocessing-sfa/internal/workflow.(*PreprocessingWorkflowResult).systemError\n\t/src/preprocessing-worker/internal/workflow/preprocessing.go:96\ngithub.com/artefactual-sdps/preprocessing-sfa/internal/workflow.(*PreprocessingWorkflow).Execute\n\t/src/preprocessing-worker/internal/workflow/preprocessing.go:209\nreflect.Value.call\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.linux-amd64/src/reflect/value.go:581\nreflect.Value.Call\n\t/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.23.2.linux-amd64/src/reflect/value.go:365\ngo.temporal.io/sdk/internal.executeFunction\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker.go:1858\ngo.temporal.io/sdk/internal.(*workflowEnvironmentInterceptor).ExecuteWorkflow\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/workflow.go:546\ngo.temporal.io/sdk/internal.(*WorkflowInboundInterceptorBase).ExecuteWorkflow\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/interceptor_base.go:154\ngo.temporal.io/sdk/internal.(*workflowExecutor).Execute\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_worker.go:811\ngo.temporal.io/sdk/internal.(*syncWorkflowDefinition).Execute.func1\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_workflow.go:528\ngo.temporal.io/sdk/internal.(*coroutineState).run\n\t/go/pkg/mod/go.temporal.io/sdk@v1.26.1/internal/internal_workflow.go:1121"}