use_dualstack_endpoint=True does not fallback to regular endpoint

[fluxth]

Describe the bug

When explicitly passing aioboto.config.Config(use_dualstack_endpoint=True) to a client:

client = boto3.client(
    "sts",
    region_name="ap-northeast-1",
    config=Config(use_dualstack_endpoint=True),
)

boto will always use https://{service}.{region}.api.aws endpoint for that service even if that service doesn't support dual stack.

Expected Behavior

use_dualstack_endpoint=True should fallback to regular endpoint when the dual stack endpoint does not exist.

For STS service, the dual stack endpoint at https://sts.ap-northeast-1.api.aws/ does not exist.
Boto should fallback to https://sts.ap-northeast-1.amazonaws.com/ endpoint in this case.

https://docs.aws.amazon.com/general/latest/gr/sts.html

Current Behavior

Boto will use non-existent dual stack endpoint for services that still doesn't support dual stack instead of falling back to the regular endpoint.

(see Reproduction Steps)

Reproduction Steps

This script should print the current STS identity.

import boto3
from botocore.config import Config

client = boto3.client(
    "sts",
    region_name="ap-northeast-1",
    config=Config(use_dualstack_endpoint=True),
)

response = client.get_caller_identity()

print(response)

But instead, we get this exception:

botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://sts.ap-northeast-1.api.aws/"

Possible Solution

/botocore/data/endpoints.json file should have sufficient tagging to detect if a service has dual stack enabled.

Additional Information/Context

No response

SDK version used

boto3 = "1.28.64"

Environment details (OS name and version, etc.)

Fedora Linux 38

[tim-finnigan]

Hi @fluxth thanks for reaching out. I raised this issue for discussion with the team and was told that the current behavior is expected. AWS SDKs like Boto3 produce the dualstack endpoint whether it exists or not when use_dualstack_endpoint=True is set. The reason is because the SDKs are forwards-compatible for this behavior. The service may release support for dualstack in the future, and users should not need to upgrade their SDK to use it.

I reached out to the STS team to get more information regarding their plans to add support for these endpoints. I can update this issue when we hear back, but given the intended design here - the request to fall back to a regular endpoint is not under consideration.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[mmerickel]

What is the recourse here? If I set AWS_USE_DUALSTACK_ENDPOINT=true in my environment, it is just literally broken when talking to STS. Is there an environment variable to set this on a per-service basis? https://docs.aws.amazon.com/sdkref/latest/guide/feature-endpoints.html does not document that you are just hosed if you use this feature and certain services don't actually support it YET.

Specifically I'd like an environment variable to enable this for S3, and maybe a couple other services, and then obviously keep STS on the default endpoints until it actually supports a dualstack endpoint. It's just amazing to me that botocore would be ok making up a DNS endpoint for STS that doesn't exist right now.

[mmerickel]

Looks like the only approach right now is per-service opt-in, for example AWS_ENDPOINT_URL_S3=https://s3.dualstack.us-east-2.amazonaws.com. The automatic dualstack negotiation via AWS_USE_DUALSTACK_ENDPOINT=true is just fundamentally unsafe - would love for this to be re-evaluated. Thanks!

[aghassemlouei]

Agree with @mmerickel regarding the re-evaluation. We'll open up an internal support case and engage our Technical Account Managers but appreciate @fluxth raising this issue!

[mmerickel]

I ran into another problem that occurs due to the brokenness of this feature. Since the only way to use a dualstack endpoint right now is to override it per-service, I can't do it in such a way that it talks to an appropriate dualstack endpoint for the correct region via external config files or env vars.

If I set AWS_ENDPOINT_URL_S3=https://s3.dualstack.us-east-2.amazonaws.com then my app no longer works properly with any buckets in any region other than us-east-2. This is unacceptable of course, and wouldn't be an issue if AWS_USE_DUALSTACK_ENDPOINT=true worked properly. Instead we have to push this logic further down into our tools and set the endpoint url manually on every client connection so that we pick the right endpoint ourselves based on which S3 bucket we are trying to connect to.

I should be able to configure this better in the environment. Right now botocore / aws decisions here are actively making it harder to use any service endpoints that support IPv6 when they happen to be available.