Skip to content

EmrStepSensor to auto detect stepid's #11877

@pedresnyman

Description

@pedresnyman

Description

Currently the EmrStepSensor only looks at one step at a time. Enhancing the EmrStepSensor to retrieve all steps on a specified cluster if no stepId was provided.

Use case / motivation

If no step_id was provided the EmrStepSensor wil retrieve all steps on a specified cluster.
This allows you to retrieve all steps and their states with a singe api call.

What happened

We ran into a throttle limit.

botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the DescribeStep operation (reached max retries: 4): Rate exceeded

With the enhancement of the EmrStepSensor that is taken care of and also then the EmrStepSensor is more dynamic.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions