AWS dms describe-endpoints filters doesn't work if source and target are provided in filter types

[divakarpatil51]

Hi,

I tried to filter dms endpoints by providing source and target as filters but it is throwing a 500 error. I am not sure if filtering is designed the way it is or a bug.

`aws dms describe-endpoints --filters Name=endpoint-type,Values=source,target

Output:
An error occurred (InternalFailure) when calling the DescribeEndpoints operation (reached max retries: 2): An internal error has occurred. Please try your query again at a later time.`

[RyanFitzSimmonsAK]

Hi @divakarpatil51, thanks for reaching out. Could you provide debug logs of this behavior? You can get debug logs by adding --debug to your command, and redacting any sensitive information. Thanks!

[divakarpatil51]

Hi @RyanFitzSimmonsAK, PFB debug logs:

Command:
aws dms describe-endpoints --filters Name=endpoint-type,Values=source,target --debug

Logs:

2023-06-01 21:13:46,425 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['dms', 'describe-endpoints', '--filters', 'Name=endpoint-type,Values=source,target', '--debug']
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_s3 at 0x7fa68a0ee4c0>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_ddb at 0x7fa689f48c10>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.configure.configure.ConfigureCommand'>>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fa689ee7ca0>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function change_name at 0x7fa689eefca0>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function alias_opsworks_cm at 0x7fa68a101e50>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_history_commands at 0x7fa689f94790>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <bound method BasicCommand.add_command of <class 'awscli.customizations.devcommands.CLIDevCommand'>>
2023-06-01 21:13:46,545 - MainThread - botocore.hooks - DEBUG - Event building-command-table.main: calling handler <function add_waiters at 0x7fa68a1010d0>
2023-06-01 21:13:46,549 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/data/cli.json
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_types at 0x7fa68a0491f0>
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function no_sign_request at 0x7fa68a049d30>
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_verify_ssl at 0x7fa68a049ca0>
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_read_timeout at 0x7fa68a049e50>
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <function resolve_cli_connect_timeout at 0x7fa68a049dc0>
2023-06-01 21:13:46,550 - MainThread - botocore.hooks - DEBUG - Event top-level-args-parsed: calling handler <built-in method update of dict object at 0x7fa68a1a0840>
2023-06-01 21:13:46,551 - MainThread - awscli.clidriver - DEBUG - CLI version: aws-cli/2.8.7 Python/3.9.11 Darwin/22.4.0 exe/x86_64 prompt/off
2023-06-01 21:13:46,551 - MainThread - awscli.clidriver - DEBUG - Arguments entered to CLI: ['dms', 'describe-endpoints', '--filters', 'Name=endpoint-type,Values=source,target', '--debug']
2023-06-01 21:13:46,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_timestamp_parser at 0x7fa68a0eeaf0>
2023-06-01 21:13:46,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function register_uri_param_handler at 0x7fa689bbe0d0>
2023-06-01 21:13:46,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function add_binary_formatter at 0x7fa68a162f70>
2023-06-01 21:13:46,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function no_pager_handler at 0x7fa689b84550>
2023-06-01 21:13:46,551 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_assume_role_provider_cache at 0x7fa689be4c10>
2023-06-01 21:13:46,555 - MainThread - botocore.utils - DEBUG - IMDS ENDPOINT: http://169.254.169.254/
2023-06-01 21:13:46,566 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function attach_history_handler at 0x7fa689f94670>
2023-06-01 21:13:46,567 - MainThread - botocore.hooks - DEBUG - Event session-initialized: calling handler <function inject_json_file_cache at 0x7fa689f44af0>
2023-06-01 21:13:46,627 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/dms/2016-01-01/service-2.json
2023-06-01 21:13:46,636 - MainThread - botocore.hooks - DEBUG - Event building-command-table.dms: calling handler <function add_waiters at 0x7fa68a1010d0>
2023-06-01 21:13:46,691 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/dms/2016-01-01/waiters-2.json
2023-06-01 21:13:46,692 - MainThread - awscli.clidriver - DEBUG - OrderedDict([('filters', <awscli.arguments.ListArgument object at 0x7fa68a574730>), ('max-records', <awscli.arguments.CLIArgument object at 0x7fa68a574760>), ('marker', <awscli.arguments.CLIArgument object at 0x7fa68a574790>)])
2023-06-01 21:13:46,692 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.dms.describe-endpoints: calling handler <function add_streaming_output_arg at 0x7fa68a0eedc0>
2023-06-01 21:13:46,692 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.dms.describe-endpoints: calling handler <function add_cli_input_json at 0x7fa689be8430>
2023-06-01 21:13:46,693 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.dms.describe-endpoints: calling handler <function add_cli_input_yaml at 0x7fa689be8670>
2023-06-01 21:13:46,694 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.dms.describe-endpoints: calling handler <function unify_paging_params at 0x7fa689f551f0>
2023-06-01 21:13:46,756 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/dms/2016-01-01/paginators-1.json
2023-06-01 21:13:46,756 - MainThread - awscli.customizations.paginate - DEBUG - Modifying paging parameters for operation: DescribeEndpoints
2023-06-01 21:13:46,756 - MainThread - botocore.hooks - DEBUG - Event building-argument-table.dms.describe-endpoints: calling handler <function add_generate_skeleton at 0x7fa68a03d790>
2023-06-01 21:13:46,757 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.dms.describe-endpoints: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fa68a5745e0>>
2023-06-01 21:13:46,757 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.dms.describe-endpoints: calling handler <bound method OverrideRequiredArgsArgument.override_required_args of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fa68a5747f0>>
2023-06-01 21:13:46,757 - MainThread - botocore.hooks - DEBUG - Event before-building-argument-table-parser.dms.describe-endpoints: calling handler <bound method GenerateCliSkeletonArgument.override_required_args of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fa68a574a30>>
2023-06-01 21:13:46,757 - MainThread - botocore.hooks - DEBUG - Event operation-args-parsed.dms.describe-endpoints: calling handler functools.partial(<function check_should_enable_pagination at 0x7fa689f55310>, ['marker', 'max-records'], {}, OrderedDict([('filters', <awscli.arguments.ListArgument object at 0x7fa68a574730>), ('max-records', <awscli.arguments.CLIArgument object at 0x7fa68a574760>), ('marker', <awscli.arguments.CLIArgument object at 0x7fa68a574790>), ('cli-input-json', <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fa68a5745e0>), ('cli-input-yaml', <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fa68a5747f0>), ('starting-token', <awscli.customizations.paginate.PageArgument object at 0x7fa68a574490>), ('page-size', <awscli.customizations.paginate.PageArgument object at 0x7fa68a5749d0>), ('max-items', <awscli.customizations.paginate.PageArgument object at 0x7fa68a574550>), ('generate-cli-skeleton', <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fa68a574a30>)]))
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.filters: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event process-cli-arg.dms.describe-endpoints: calling handler <awscli.argprocess.ParamShorthandParser object at 0x7fa689baba90>
2023-06-01 21:13:46,758 - MainThread - awscli.argprocess - DEBUG - Parsing param --filters as shorthand
2023-06-01 21:13:46,758 - MainThread - awscli.arguments - DEBUG - Unpacked value of ['Name=endpoint-type,Values=source,target'] for parameter "filters": [{'Name': 'endpoint-type', 'Values': ['source', 'target']}]
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.max-records: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.marker: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.cli-input-json: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.cli-input-yaml: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.starting-token: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.page-size: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.max-items: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event load-cli-arg.dms.describe-endpoints.generate-cli-skeleton: calling handler <awscli.paramfile.URIArgumentHandler object at 0x7fa68a1e5f70>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event calling-command.dms.describe-endpoints: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputJSONArgument object at 0x7fa68a5745e0>>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event calling-command.dms.describe-endpoints: calling handler <bound method CliInputArgument.add_to_call_parameters of <awscli.customizations.cliinput.CliInputYAMLArgument object at 0x7fa68a5747f0>>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event calling-command.dms.describe-endpoints: calling handler <bound method GenerateCliSkeletonArgument.generate_skeleton of <awscli.customizations.generatecliskeleton.GenerateCliSkeletonArgument object at 0x7fa68a574a30>>
2023-06-01 21:13:46,758 - MainThread - botocore.hooks - DEBUG - Event calling-command.dms.describe-endpoints: calling handler functools.partial(<function check_should_enable_pagination_call_parameters at 0x7fa689f55700>, ['Marker', 'MaxRecords'])
2023-06-01 21:13:46,759 - MainThread - botocore.credentials - DEBUG - Looking for credentials via: env
2023-06-01 21:13:46,759 - MainThread - botocore.credentials - INFO - Found credentials in environment variables.
2023-06-01 21:13:46,764 - MainThread - botocore.loaders - DEBUG - Loading JSON file: /usr/local/aws-cli/awscli/botocore/data/endpoints.json
2023-06-01 21:13:46,779 - MainThread - botocore.hooks - DEBUG - Event choose-service-name: calling handler <function handle_service_name_alias at 0x7fa688a06430>
2023-06-01 21:13:46,781 - MainThread - botocore.hooks - DEBUG - Event creating-client-class.database-migration-service: calling handler <function add_generate_presigned_url at 0x7fa68886d1f0>
2023-06-01 21:13:46,906 - MainThread - botocore.endpoint - DEBUG - Setting dms timeout as (60, 60)
2023-06-01 21:13:46,907 - MainThread - botocore.hooks - DEBUG - Event provide-client-params.database-migration-service.DescribeEndpoints: calling handler <function base64_decode_input_blobs at 0x7fa68a164700>
2023-06-01 21:13:46,907 - MainThread - botocore.hooks - DEBUG - Event before-parameter-build.database-migration-service.DescribeEndpoints: calling handler <function generate_idempotent_uuid at 0x7fa688a20550>
2023-06-01 21:13:46,908 - MainThread - botocore.hooks - DEBUG - Event before-call.database-migration-service.DescribeEndpoints: calling handler <function inject_api_version_header_if_needed at 0x7fa688a25dc0>
2023-06-01 21:13:46,908 - MainThread - botocore.endpoint - DEBUG - Making request for OperationModel(name=DescribeEndpoints) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonDMSv20160101.DescribeEndpoints', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'aws-cli/2.8.7 Python/3.9.11 Darwin/22.4.0 exe/x86_64 prompt/off command/dms.describe-endpoints'}, 'body': b'{"Filters": [{"Name": "endpoint-type", "Values": ["source", "target"]}]}', 'url': '', 'context': {'client_region': '', 'client_config': <botocore.config.Config object at 0x7fa68aa9cd90>, 'has_streaming_input': False, 'auth_type': None}}
2023-06-01 21:13:46,908 - MainThread - botocore.hooks - DEBUG - Event request-created.database-migration-service.DescribeEndpoints: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7fa68aa9ce50>>
2023-06-01 21:13:46,908 - MainThread - botocore.hooks - DEBUG - Event choose-signer.database-migration-service.DescribeEndpoints: calling handler <function set_operation_specific_signer at 0x7fa688a20430>
2023-06-01 21:13:46,909 - MainThread - botocore.auth - DEBUG - Calculating signature using v4 auth.
2023-06-01 21:13:46,909 - MainThread - botocore.auth - DEBUG - CanonicalRequest:
POST
/

content-type:application/x-amz-json-1.1
host:dms.eu-west-1.amazonaws.com
x-amz-date:20230601T111346Z

content-type;host;x-amz-date;x-amz-security-token;x-amz-target
2023-06-01 21:13:46,909 - MainThread - botocore.auth - DEBUG - StringToSign:
2023-06-01 21:13:46,909 - MainThread - botocore.auth - DEBUG - Signature:
2023-06-01 21:13:46,910 - MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://dms.eu-west-1.amazonaws.com/, headers={'X-Amz-Target': b'AmazonDMSv20160101.DescribeEndpoints', 'Content-Type': b'application/x-amz-json-1.1', 'User-Agent': b'aws-cli/2.8.7 Python/3.9.11 Darwin/22.4.0 exe/x86_64 prompt/off command/dms.describe-endpoints', 'X-Amz-Date': b'20230601T111346Z', 'X-Amz-Security-Token': b'', 'Authorization': b'', 'Content-Length': '72'}>
2023-06-01 21:13:46,910 - MainThread - botocore.httpsession - DEBUG - Certificate path: /usr/local/aws-cli/awscli/botocore/cacert.pem
2023-06-01 21:13:46,910 - MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): dms.eu-west-1.amazonaws.com:443
2023-06-01 21:13:48,593 - MainThread - urllib3.connectionpool - DEBUG - https://dms.eu-west-1.amazonaws.com:443 "POST / HTTP/1.1" 500 117
2023-06-01 21:13:48,593 - MainThread - botocore.parsers - DEBUG - Response headers: {'x-amzn-RequestId': 'dd4cb123-28fa-48be-9f55-01eaa3940ca1', 'Date': 'Thu, 01 Jun 2023 11:13:49 GMT', 'Content-Type': 'application/x-amz-json-1.1', 'Content-Length': '117'}
2023-06-01 21:13:48,594 - MainThread - botocore.parsers - DEBUG - Response body:
b'{"__type":"InternalFailure","message":"An internal error has occurred. Please try your query again at a later time."}'
2023-06-01 21:13:48,596 - MainThread - botocore.hooks - DEBUG - Event needs-retry.database-migration-service.DescribeEndpoints: calling handler <bound method RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fa68aae2910>>
2023-06-01 21:13:48,596 - MainThread - botocore.retries.standard - DEBUG - Retry needed, retrying request after delay of: 0.8583243367684047
 RetryHandler.needs_retry of <botocore.retries.standard.RetryHandler object at 0x7fa68aae2910>>
2023-06-01 21:13:50,365 - MainThread - botocore.retries.standard - DEBUG - Max attempts of 3 reached.
2023-06-01 21:13:50,366 - MainThread - botocore.retries.standard - DEBUG - Not retrying request.
2023-06-01 21:13:50,366 - MainThread - botocore.hooks - DEBUG - Event after-call.database-migration-service.DescribeEndpoints: calling handler <bound method RetryQuotaChecker.release_retry_quota of <botocore.retries.standard.RetryQuotaChecker object at 0x7fa68aae2340>>
2023-06-01 21:13:50,366 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
  File "awscli/clidriver.py", line 458, in main
  File "awscli/clidriver.py", line 593, in __call__
  File "awscli/clidriver.py", line 769, in __call__
  File "awscli/clidriver.py", line 902, in invoke
  File "awscli/clidriver.py", line 924, in _display_response
  File "awscli/formatter.py", line 82, in __call__
  File "awscli/botocore/paginate.py", line 446, in build_full_result
  File "awscli/botocore/paginate.py", line 252, in __iter__
  File "awscli/botocore/paginate.py", line 329, in _make_request
  File "awscli/botocore/client.py", line 309, in _api_call
  File "awscli/botocore/client.py", line 639, in _make_api_call
botocore.exceptions.ClientError: An error occurred (InternalFailure) when calling the DescribeEndpoints operation (reached max retries: 2): An internal error has occurred. Please try your query again at a later time.

An error occurred (InternalFailure) when calling the DescribeEndpoints operation (reached max retries: 2): An internal error has occurred. Please try your query again at a later time.```

[RyanFitzSimmonsAK]

Hi @divakarpatil51, thanks for following up. It looks like the endpoint-type filter is intended to filter either source or target, rather than both at the same time. Both of the following commands worked for me.

aws dms describe-endpoints --filters Name=endpoint-type,Values=source
aws dms describe-endpoints --filters Name=endpoint-type,Values=target

Since endpoints are either a source or a target, filtering both of them would be the same as filtering neither of them. I hope this answers your question, please let me know if you have any follow-up questions.

@github-actions proposed answer