Sporadic Issues with Boto3 in a Python Microservice Accessing S3

[itburkealert]

We have a micro service running in EKS that is using the following specifications:

boto3 s3 client, version 1.28.71
base image: python:3.7.11-bullseye

The service has worked smoothly for several months and we recently started observing an occassional issue in which the service begins throwing the following error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/s3_storage/S3Storage.py", line 65, in get_file
    self._s3_client.download_file(bucket_name, path_to_obj, file_path)
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/inject.py", line 197, in download_file
    callback=Callback,
  File "/usr/local/lib/python3.7/site-packages/boto3/s3/transfer.py", line 405, in download_file
    future.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 103, in result
    return self._coordinator.result()
  File "/usr/local/lib/python3.7/site-packages/s3transfer/futures.py", line 266, in result
    raise self._exception
  File "/usr/local/lib/python3.7/site-packages/s3transfer/tasks.py", line 269, in _main
    self._submit(transfer_future=transfer_future, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/s3transfer/download.py", line 357, in _submit
    **transfer_future.meta.call_args.extra_args,
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 553, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 1009, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

This error is also presented in the following link

Our S3 credentials are confirmed to be good and we haven't needed to modify this microservice's code in some time. We understand that boto3 has dropped support for python 3.7 so we're curious if that's what's causing the issue here.

Any additional insight that can be offered would be helpful.

[RyanFitzSimmonsAK]

Hey @itburkealert, thanks for reaching out. Since this is a Boto3 specific issue, I'm going to close this issue and create a duplicate in the Boto3 repository. Thanks!

boto/boto3#4277

[github-actions]

This issue is now closed.

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.