unhashable type: 'bytearray' when using socks5 proxy

[Vigofs]

https://github.com/encode/httpx/discussions

Gives the error unhashable type: 'bytearray' when I make a GET request using cookies, headers and parameters

Code:

for _ in range(retries):
    response = await self.session.request(
                method=method, url=url, data=data, params=params,headers=headers,
                follow_redirects=allow_redirects, timeout=35
            )

[Vigofs]

and i am use proxy (socks5)

[Vigofs]

data and params is none

[jhominal]

In order to say anything about your issue, we are going to need more information.

Could you:

• Give us the stack trace that accompanies the error?
• Inspect and print the contents of all the variables that you pass (particularly, headers)?

It would be tremendously helpful if you were able to write a runnable reproduction of the issue you have.

[JamzumSum]

@jhominal I've met the same error, MWE as below

from httpx import AsyncClient

async def test_socks5_proxy():
    async with AsyncClient(proxies="socks5://localhost:7890") as client:
        r = await client.post(f"https://api.telegram.org/bot{token}/getMe")  # TypeError
        r.raise_for_status()

Stack trace:

  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\test\test_httpx_socks.py", line 13, in <module>
    asyncio.run(test_socks5_proxy())
  File "D:\Pythons\3.10\lib\asyncio\runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "D:\Pythons\3.10\lib\asyncio\base_events.py", line 646, in run_until_complete
    return future.result()
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\test\test_httpx_socks.py", line 9, in test_socks5_proxy    
    r = await client.post(f"https://api.telegram.org/bot{token}/getMe")
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1842, in post
    return await self.request(
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1614, in send
    response = await self._send_handling_auth(
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1642, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1679, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_client.py", line 1716, in _send_single_request
    response = await transport.handle_async_request(request)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpx\_transports\default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request
    raise exc
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpcore\_async\socks_proxy.py", line 290, in handle_async_request
    raise exc
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpcore\_async\socks_proxy.py", line 238, in handle_async_request
    await _init_socks5_connection(**kwargs)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\httpcore\_async\socks_proxy.py", line 93, in _init_socks5_connection
    response = conn.receive_data(incoming_bytes)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\socksio\socks5.py", line 382, in receive_data
    reply = SOCKS5Reply.loads(data)
  File "e:\Documents\CODE_ROOT\aioqzone\Qzone3TG\.venv\lib\site-packages\socksio\socks5.py", line 262, in loads
    addr=decode_address(AddressType.from_socks5_atype(atype), data[4:-2]),
TypeError: unhashable type: 'bytearray'

• httpx 0.23.0
• socksio 1.0.0
• windows 11
• Python 3.10.4 (tags/v3.10.4:9d38120, Mar 23 2022, 23:13:41) [MSC v.1929 64 bit (AMD64)]
• socks proxy is okay, unless my browser uses it as usual

Downstream issue and discussions here, if you need it: python-telegram-bot/python-telegram-bot#3211

[DarioDaF]

Can confirm the error, happens always when using async and socks5, data is received in bytearray form down to socksio which supposes it's bytes, usually this wouldn't be an issue but decode_address is cached by @functools.lru_cache that tries to hash the function arguments aka those bytearray failing to do so, I'm looking at a fix atm and can potentially PR if it's in my capabilities

[DarioDaF]

I have a very sad workaround: in _init_socks5_connection wrap all the await stream.read calls in a cast to bytes, it's preatty confusing seeing that the "wrong type" goes down to anyio and possibly even further, not sure if it makes sense to patch it that way, if it's ok I can do a PR, but will still look at the source if I can find it

[JamzumSum]

I made a PR at sethmlarson/socksio#58. Currently it is not merged. I'm using the branch as a substitution of socksio, and my code works fine.

[DarioDaF]

very good although it's probably not intended at the root since all prototype name bytes, I also opened an issue for asyncio since it's there the cause python/cpython#99941

[JamzumSum]

I agree it is an asyncio issue. I thought it is bounded to windows ProactorEventLoop.
(tested 3 months ago. i've kind of forgotten)

[tomchristie]

If needed we can fix/workaround this from our side, by making sure we coerce the incoming data to bytes.

Eg here... https://github.com/encode/httpcore/blob/master/httpcore/_async/socks_proxy.py#L58

[DarioDaF]

Probably it will not be needed, they are going through In the socket library with a workaround for py 3.10 and 3.11 and will probably be fixed up in the next python release

[Raichuu41]

Are there any new updates to it? The problem is still there unfortunately

[DarioDaF]

Ok tested and can confirm python/cpython#99941 is still present even in Python 3.11.3 I don't know when the fix will be actually released, for version info probably asking in the mentioned issue is the best, otherwise you could potentially install JamzumSum branch of socksio mentioned above in the comments that has a workaround

[Raichuu41]

Thank you for confirming the issue. I hope it will be fixed soon. For now, I just use the HTTP proxies instead of SOCKS5. With HTTP proxies, everything seems to work fine so far. It's not a long-term solution but currently I don't rely on the socks5 usage of proxies in Windows as of now.
So if anyone encounters the same issue and cannot switch to Linux, HTTP proxy may be an alternative.

[tomchristie]

I reckon we'd be open to pull requests resolving this socks5 issue sorted in httpcore, even if it kinda is a Python issue really.

If anyone wants to handle that, then escalating this into an issue on the httpcore repo would be a good first step.

[auqhjjqdo]

I tried using another socks library and it works fine with python 3.10.11, maybe consider replacing the socksio dependency with this
https://github.com/romis2012/httpx-socks

[tomchristie]

We could certainly at least link to that package in the docs...

https://www.python-httpx.org/third_party_packages/