Test regressions with urllib3-2.2.0: botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes found

[mgorny]

After upgrading urllib3 to 2.2.0, a few tests fail (full log below):

FAILED tests/test_moto_api/recorder/test_recorder.py::TestThreadedMotoServer::test_server - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs0-200-200] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs1-200-200] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs2-200-403] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs3-200-200] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs4-403-403] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_or_allow_get_object[kwargs5-200-403] - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_put_object - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_all_actions - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...
ERROR tests/test_s3/test_s3_bucket_policy.py::TestBucketPolicy::test_block_all_with_different_principal - botocore.exceptions.HTTPClientError: An HTTP Client raised an unhandled exception: sequence item 0: expected str instance, bytes f...

To reproduce:

pip install urllib3==2.2.0
python -m pytest tests/test_s3/test_s3_bucket_policy.py tests/test_moto_api/recorder/test_recorder.py

Full log (400k): log.txt

[bblommers]

Hi @mgorny. The stacktrace shows that it fails when botocore sends a HTTP request, and they indeed do not support urllib3>=2 yet. So I'm inclined to assume that's the reason Moto's tests fail.

[mgorny]

While they formally don't declare support, it works just fine. As the bug indicates, they are concerned about removing the pin as they're afraid it could break assumptions reverse dependencies made. moto works fine with urllib3 up to 2.1.0, and awscli+boto3+botocore works fine and passes all tests with 2.2.0.

[bblommers]

The following test fails with the same problem, without Moto:

import boto3
from uuid import uuid4


def test():
    b = str(uuid4())
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket=b)
    try:
        s3.put_object(Bucket=b, Body=b"ABCD", Key="data")
        s3.delete_object(Bucket=b, Key="data")
    finally:
        s3.delete_bucket(Bucket=b)

More specifically, it fails on this line, which is before the request is actually executed - and therefore before it could reach Moto.

Relevant issue in botocore: boto/botocore#3111

[rkmish]

@bblommers & @mgorny - which version of botocore,boto3 and urllib3 I should install to overcome this http issue , could you please help.

[bblommers]

@rkmish Any version of botocore/boto3, but urllib3<=2.1.0 max

[bblommers]

@mgorny @rkmish urllib3 has released a new version, 2.2.1, which contains a fix for this problem. All Moto's internal tests pass with this version installed, so I think this is resolved.

[mgorny]

Yes, I've merged that patch to Gentoo as soon as it was committed. Thank you!