Add additional waiter config validation for matching error codes #3253
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
We should validate that if an error matcher references an error code, then the operation actually has a modeled error with a matching code. This won't pass as is, so I'm adding this as a reference for now. We likely won't be able to get all the service models updated immediately, so we may need to allow list existing cases that are valid but not fully modeled.
jamesls
force-pushed
the
waiter-error-validate
branch
from
9d51bce
to
74745b9
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We should validate that if an error matcher references an error code, then the operation actually has a modeled error with a matching code.
This won't pass as is, so I'm adding this as a reference for the time being. We likely won't be able to get all the service models updated immediately, so we may need to allow list existing cases that are valid but not fully modeled if you do want to pull this in.
Side note, looks like when this file was migrated to pytest a few years ago (#2495) the parameterization logic was changed such that instead of 1 test per individual waiter, it's now 1 test for all waiters within a service. This will still catch any issues, it just means that it'll fail at the first waiter validation error is encounters in the service so there may actually be more waiters that are invalid for the service.
Results
Looks like there are a few additional failures, with a mix of error and unmodeled behavior:
AuditReportCreatedwaiter for theacm-pcaservice that definesa matcher with error code AccessDeniedException that's not in the error codes...ChangeSetCreateCompletewaiter for thecloudformationservice that definesa matcher with error code ValidationError that's not in the error...ImageExistswaiter for theec2service that definesa matcher with error code InvalidAMIID.NotFound that's not in the error codes defined in...LoadBalancerAvailablewaiter for theelbv2service that definesa matcher with error code LoadBalancerNotFoundException that's not in the er...TenantDatabaseDeletedwaiter for therdsservice that definesa matcher with error code DBInstanceNotFoundFault that's not in the error code...EndpointDeletedwaiter for thesagemakerservice that definesa matcher with error code ValidationException that's not in the error codes de...Looks like there's two errors, one's being tracked, and one from rds that's not.