-
Notifications
You must be signed in to change notification settings - Fork 16.7k
Closed
Labels
good first issuekind:featureFeature RequestsFeature Requestsprovider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues
Description
Description
Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.
Use case / motivation
If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.
What happened
We ran into a throttle limit.
botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded
With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
good first issuekind:featureFeature RequestsFeature Requestsprovider:amazonAWS/Amazon - related issuesAWS/Amazon - related issues