Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempting to fetch credentials via iam-role (AWS SSO) is failing due to leading to an unreachable host #3030

Closed
konatacarneiro opened this issue Oct 3, 2023 · 7 comments
Assignees
Labels
bug This issue is a confirmed bug. closing-soon credentials p2 This is a standard priority issue response-requested Waiting on additional info and feedback.

Comments

@konatacarneiro
Copy link

Describe the bug

When trying to get credentials using AWS SSO, triggering the iam-role credentials provider, no credentials are returned. Later, when boto3 tries to actually use the credentials, since no credentials exist, a botocore.exceptions.NoCredentialsError is obtained.

Expected Behavior

Valid credentials should be obtained through the iam-role credentials provider via AWS SSO, just like it's working in other languages currently (e.g. I tested it in Go via aws-sdk-go).

Current Behavior

The provider seems to start a connection to http://169.254.169.254:80 to attempt to fetch a token, but that host is unreachable.

Debug output:

DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway
DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
DEBUG:botocore.hooks:Changing event name from docs.*.logs.CreateExportTask.complete-section to docs.*.cloudwatch-logs.CreateExportTask.complete-section
DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
DEBUG:botocore.credentials:Looking for credentials via: env
DEBUG:botocore.credentials:Looking for credentials via: assume-role
DEBUG:botocore.credentials:Looking for credentials via: assume-role-with-web-identity
DEBUG:botocore.credentials:Looking for credentials via: shared-credentials-file
DEBUG:botocore.credentials:Looking for credentials via: custom-process
DEBUG:botocore.credentials:Looking for credentials via: config-file
DEBUG:botocore.credentials:Looking for credentials via: ec2-credentials-file
DEBUG:botocore.credentials:Looking for credentials via: boto-config
DEBUG:botocore.credentials:Looking for credentials via: container-role
DEBUG:botocore.credentials:Looking for credentials via: iam-role
DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 169.254.169.254:80
DEBUG:botocore.utils:Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/api/token: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 158, in _new_conn
    conn = connection.create_connection(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 254, in send
    urllib_response = conn.urlopen(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 637, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 91, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 119, in _send_output
    self.send(msg)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 203, in send
    return super(AWSConnection, self).send(str)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 162, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f5307b92290>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/utils.py", line 296, in _fetch_metadata_token
    response = self._session.send(request.prepare())
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 287, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/api/token"
DEBUG:urllib3.connectionpool:Starting new HTTP connection (2): 169.254.169.254:80
DEBUG:botocore.utils:Caught retryable HTTP exception while making metadata service request to http://169.254.169.254/latest/meta-data/iam/security-credentials/: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 158, in _new_conn
    conn = connection.create_connection(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 80, in create_connection
    raise err
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/connection.py", line 70, in create_connection
    sock.connect(sa)
TimeoutError: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 254, in send
    urllib_response = conn.urlopen(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 637, in urlopen
    retries = retries.increment(method, url, error=e, _pool=self,
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/util/retry.py", line 344, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/packages/six.py", line 686, in reraise
    raise value
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 597, in urlopen
    httplib_response = self._make_request(conn, method, url,
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 91, in _send_request
    rval = super(AWSConnection, self)._send_request(
  File "/usr/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 119, in _send_output
    self.send(msg)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/awsrequest.py", line 203, in send
    return super(AWSConnection, self).send(str)
  File "/usr/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 181, in connect
    conn = self._new_conn()
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/urllib3/connection.py", line 162, in _new_conn
    raise ConnectTimeoutError(
urllib3.exceptions.ConnectTimeoutError: (<botocore.awsrequest.AWSHTTPConnection object at 0x7f5307b93400>, 'Connection to 169.254.169.254 timed out. (connect timeout=1)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/utils.py", line 339, in _get_request
    response = self._session.send(request.prepare())
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/httpsession.py", line 287, in send
    raise ConnectTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ConnectTimeoutError: Connect timeout on endpoint URL: "http://169.254.169.254/latest/meta-data/iam/security-credentials/"
DEBUG:botocore.utils:Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.

DEBUG:botocore.loaders:Loading JSON file: /home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/data/endpoints.json
DEBUG:botocore.hooks:Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f5307cc2440>
DEBUG:botocore.loaders:Loading JSON file: /home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/data/secretsmanager/2017-10-17/service-2.json
DEBUG:botocore.loaders:Loading JSON file: /home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/data/secretsmanager/2017-10-17/service-2.sdk-extras.json
DEBUG:botocore.hooks:Event creating-client-class.secrets-manager: calling handler <function add_generate_presigned_url at 0x7f5307cad360>
DEBUG:botocore.endpoint:Setting secretsmanager timeout as (60, 60)
DEBUG:botocore.loaders:Loading JSON file: /home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/data/_retry.json
DEBUG:botocore.client:Registering retry handlers for service: secretsmanager
DEBUG:botocore.hooks:Event before-parameter-build.secrets-manager.GetSecretValue: calling handler <function generate_idempotent_uuid at 0x7f5307ce56c0>
DEBUG:botocore.hooks:Event before-call.secrets-manager.GetSecretValue: calling handler <function inject_api_version_header_if_needed at 0x7f5307ce7130>
DEBUG:botocore.endpoint:Making request for OperationModel(name=GetSecretValue) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'secretsmanager.GetSecretValue', 'Content-Type': 'application/x-amz-json-1.1', 'User-Agent': 'Boto3/1.11.5 Python/3.10.12 Linux/6.2.0-33-generic Botocore/1.14.17'}, 'body': b'{"SecretId": "DatasetupDbPassword"}', 'url': 'https://secretsmanager.us-east-1.amazonaws.com/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f5307b93910>, 'has_streaming_input': False, 'auth_type': None}}
DEBUG:botocore.hooks:Event request-created.secrets-manager.GetSecretValue: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f5307b93760>>
DEBUG:botocore.hooks:Event choose-signer.secrets-manager.GetSecretValue: calling handler <function set_operation_specific_signer at 0x7f5307ce55a0>
Traceback (most recent call last):
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/aws_auth_test.py", line 50, in <module>
    main()
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/aws_auth_test.py", line 46, in main
    secret_value = get_secret(secret_name, region_name)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/aws_auth_test.py", line 24, in get_secret
    get_secret_value_response = client.get_secret_value(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/client.py", line 572, in _make_api_call
    http, parsed_response = self._make_request(
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/client.py", line 592, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/endpoint.py", line 115, in create_request
    self._event_emitter.emit(event_name, request=request,
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/home/bcarneiro/dock/softwares/desenv/test-aws-sso-issue/.venv/lib/python3.10/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

Reproduction Steps

The code example below attempts to fetch a secret from Secrets Manager, with debug logs for the core boto libs enabled.

You do need to have AWS IAM Identity Center set up in your account in order to reproduce this.

The environment variables needed to trigger AWS SSO authentication, as well as the secret name and AWS region, are set at the beginning of the code.

The code does gracefully handle the case where the secret does not exist, if authentication happens to be successful.

import boto3
import botocore
import logging
import os

os.environ['AWS_SDK_LOAD_CONFIG'] = '1'
os.environ['AWS_PROFILE'] = 'default'
secret_name = "TestSecret"
region_name = "us-east-1"

# Function to fetch a secret from Secrets Manager, given the secret name and AWS region
def get_secret(secret_name, region_name):
    # Create a Secrets Manager client
    session = boto3.session.Session()
    client = session.client(
        service_name='secretsmanager',
        region_name=region_name
    )

    get_secret_value_response = client.get_secret_value(
        SecretId=secret_name
    )
    return get_secret_value_response['SecretString']

def main():
    logging.basicConfig(level=logging.INFO)
    logging.getLogger("boto3").setLevel(logging.DEBUG)
    logging.getLogger("botocore").setLevel(logging.DEBUG)
    logging.getLogger("urllib3").setLevel(logging.DEBUG)

    try:
        secret_value = get_secret(secret_name, region_name)
        print(f"Secret value for '{secret_name}': {secret_value}")
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == 'ResourceNotFoundException':
            print(f"Secret '{secret_name}' not found")
        else:
            raise e

if __name__ == "__main__":
    main()

Possible Solution

There might be some issue with how the host for the token generation is being obtained. I honestly have no idea about why this issue is happening, but it clearly doesn't seem to be a widespread issue (e.g. a widespread outage at the token generation service) as it doesn't happen on other implementations of the AWS SDK (i.e. aws-sdk-go on Go).

Additional Information/Context

This is a regression. It didn't use to happen in a prior version. Unfortunately I'm not aware of which exact version that is since I've updated botocore long before I've encountered this issue.

SDK version used

botocore v1.14.17

Environment details (OS name and version, etc.)

Ubuntu 22.04.1, Python 3.10.12, boto3 v1.11.5, urllib3 v1.24.3

@konatacarneiro konatacarneiro added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Oct 3, 2023
@konatacarneiro konatacarneiro changed the title Attempting to credentials via iam-role (AWS SSO) is failing due to leading to an unreachable host Attempting to fetch credentials via iam-role (AWS SSO) is failing due to leading to an unreachable host Oct 3, 2023
@RyanFitzSimmonsAK RyanFitzSimmonsAK self-assigned this Oct 4, 2023
@RyanFitzSimmonsAK RyanFitzSimmonsAK added credentials p2 This is a standard priority issue and removed needs-triage This issue or PR still needs to be triaged. labels Oct 4, 2023
@RyanFitzSimmonsAK
Copy link
Contributor

Hi @b-carneiro, thanks for reaching out. Could you tell me more about how you're setting up AWS SSO and retrieving your credentials? Are you able to successfully log in to your SSO profile? I'd also recommend updating your versions of Boto3 and Botocore, as they are very out of date. Thanks!

@RyanFitzSimmonsAK RyanFitzSimmonsAK added response-requested Waiting on additional info and feedback. and removed bug This issue is a confirmed bug. labels Oct 4, 2023
@github-actions
Copy link

Greetings! It looks like this issue hasn’t been active in longer than five days. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

@konatacarneiro
Copy link
Author

konatacarneiro commented Oct 10, 2023

Hi @RyanFitzSimmonsAK, I should have given further clarification about the botocore and boto3 versions I listed. They were from a virtualenv in which I was trying to run an application with which I was having this issue.

In hindsight, I should have tested this outside the virtualenv. I thought I had. Because yeah, I just did so, and it works fine. It was a problem with that old version of botocore, indeed, and it doesn't happen in the newest version. So I think we can close this issue.

@github-actions
Copy link

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

@RyanFitzSimmonsAK RyanFitzSimmonsAK added the bug This issue is a confirmed bug. label Nov 2, 2023
@karkir0003
Copy link

@RyanFitzSimmonsAK I'm getting a similar error where I setup my IAM user and access through IAM identity center. Even after doing aws sso login --profile <profileName> and then running boto script, I'm getting "NoCredentialsFound error". Is there any docs for using boto3 client with AWS SSO?

@RyanFitzSimmonsAK
Copy link
Contributor

Sure, here is some documentation that might be useful.

@karkir0003
Copy link

Thank you @RyanFitzSimmonsAK. I think I was able to resolve the issue. I needed to do aws configure sso and then do export AWS_PROFILE=<sso-profile-name> and then run the script using boto3 (without any code changes)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This issue is a confirmed bug. closing-soon credentials p2 This is a standard priority issue response-requested Waiting on additional info and feedback.
Projects
None yet
Development

No branches or pull requests

3 participants