Properly handle ResourceNotReady returned by WaitUntilTasksRunningWithContext

[mieliespoor]

What would be the best way to handle a ResourceNotReady error response from WaitUntilTasksRunningWithContext?

aws-sdk-go/service/ecs/waiters.go

Line 140 in d9ed7d6

func (c *ECS) WaitUntilTasksRunningWithContext(ctx aws.Context, input *DescribeTasksInput, opts ...request.WaiterOption) error {

error waiting AWS ECS Task "arn:aws:ecs:eu-west-1:123456789012:task/ecs-cluster/22452be490a149e781a596a7847dd27c" to be in "Running" state: ResourceNotReady: failed waiting for successful resource state

I have noticed that the task does start and appear in the AWS Console, but we get a resource not ready error returned here. Would changing the WaitingOption have any impact here? I doubt it looking at it. Should a simple retry happen after waiting for a couple of seconds?

What would the best approach be here?

[RanVaknin]

Hi @mieliespoor ,

I don't see your actual code so this is just some speculation.

1. I would override the timeout of the waiter to wait longer.
2. I would check if the retryer is being called and if not, force the retryer to retry on a ResourceNotReady error.

If you could add a concise runnable code snippet, I'll take a closer look.

Thanks,
Ran~

[mieliespoor]

Hi Ran,

This is the entire method:

func (a *awsFargate) WaitUntilTaskRunning(ctx context.Context, taskARN string, cluster string, taskWaitConfig TaskWaitConfig) error {
	err := a.errIfNotInitialized()
	if err != nil {
		return fmt.Errorf("could not wait AWS Fargate task: %w", err)
	}

	a.logger.
		WithField("task-arn", taskARN).
		Debug(`[WaitUntilTaskRunning] Will wait until Fargate task is in "Running" state`)

	input := ecs.DescribeTasksInput{
		Cluster: &cluster,
		Tasks:   []*string{&taskARN},
	}

	err = a.ecsSvc.WaitUntilTasksRunningWithContext(ctx, &input, func(w *request.Waiter) {
		w.MaxAttempts = taskWaitConfig.MaxAttempts
		w.Delay = request.ConstantWaiterDelay(time.Duration(taskWaitConfig.WaitTimeout) * time.Second)
	})
	if err != nil {
		return fmt.Errorf(`error waiting AWS Fargate Task %q to be in "Running" state: %w`, taskARN, err)
	}

	a.logger.
		WithField("task-arn", taskARN).
		Debug(`[WaitUntilTaskRunning] Fargate Task in "Running" state`)

	return nil
}

[mieliespoor]

This was a blind stab at introducing the waiter and hoping it will make a change, but I'm not sure that it did anything.

Like I said, I'm very new to golang, so not sure I approached this in the correct way, or even if this will solve the issue we see.

[mieliespoor]

An earlier version of the code can be seen here. https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/fargate/-/blob/master/aws/fargate.go#L213

We see this happen fairly regularly, even when trying to start tasks on EC2 launch types. Do however note that this code linked won't start EC2 tasks as that piece of code is in a private repository. I do however hope that this will help in some way.

[mieliespoor]

on your second point, how would that work?