Skip to content

Commit 5dd7d8f

Browse files
authored
🐛 fix exception leak from urllib3-future when resolving lazy responses (#163)
fix #162
1 parent 1933840 commit 5dd7d8f

File tree

6 files changed

+189
-43
lines changed

6 files changed

+189
-43
lines changed

.pre-commit-config.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ repos:
2323
# Run the formatter.
2424
- id: ruff-format
2525
- repo: https://github.com/pre-commit/mirrors-mypy
26-
rev: v1.11.2
26+
rev: v1.12.1
2727
hooks:
2828
- id: mypy
2929
args: [--check-untyped-defs]

HISTORY.md

+6
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,12 @@
11
Release History
22
===============
33

4+
3.10.1 (2024-10-22)
5+
------------------
6+
7+
**Fixed**
8+
- Exception leak from urllib3-future when gathering / resolving lazy responses.
9+
410
3.10.0 (2024-10-21)
511
------------------
612

README.md

+39-37
Original file line numberDiff line numberDiff line change
@@ -12,41 +12,41 @@ Niquests, is the “**Safest**, **Fastest[^10]**, **Easiest**, and **Most advanc
1212
<details>
1313
<summary>👆 <b>Look at the feature table comparison</b> against <i>requests, httpx and aiohttp</i>!</summary>
1414

15-
| Feature | niquests | requests | httpx | aiohttp |
16-
|-------------------------------------|:--------------:|:---------:|:-------------:|---------------|
17-
| `HTTP/1.1` |||||
18-
| `HTTP/2` |||[^7] ||
19-
| `HTTP/3 over QUIC` |||||
20-
| `Synchronous` |||| _N/A_[^1] |
21-
| `Asynchronous` |||||
22-
| `Thread Safe` |||[^5] | _N/A_[^1] |
23-
| `Task Safe` || _N/A_[^2] |||
24-
| `OS Trust Store` |||||
25-
| `Multiplexing` ||| _Limited_[^3] ||
26-
| `DNSSEC` |[^11] ||||
27-
| `Customizable DNS Resolution` |||||
28-
| `DNS over HTTPS` |||||
29-
| `DNS over QUIC` |||||
30-
| `DNS over TLS` |||||
31-
| `Multiple DNS Resolver` |||||
32-
| `Network Fine Tuning & Inspect` ||| _Limited_[^6] | _Limited_[^6] |
33-
| `Certificate Revocation Protection` |||||
34-
| `Session Persistence` |||||
35-
| `In-memory Certificate CA & mTLS` ||| _Limited_[^4] | _Limited_[^4] |
36-
| `SOCKS 4/5 Proxies` |||||
37-
| `HTTP/HTTPS Proxies` |||||
38-
| `TLS-in-TLS Support` |||||
39-
| `Direct HTTP/3 Negotiation` |[^9] | N/A[^8] | N/A[^8] | N/A[^8] |
40-
| `Happy Eyeballs` |||||
41-
| `Package / SLSA Signed` |||||
42-
| `HTTP/2 with prior knowledge (h2c)` |||||
43-
| `Post-Quantum Security` | _Limited_[^12] ||||
44-
| `HTTP Trailers` |||||
45-
| `Early Responses` |||||
46-
| `WebSocket over HTTP/1` ||[^14] |[^14] ||
47-
| `WebSocket over HTTP/2 and HTTP/3` |[^13] ||||
48-
| `Automatic Ping for HTTP/2+` || N/A || N/A |
49-
| `Automatic Connection Upgrade / Downgrade` || N/A || N/A |
15+
| Feature | niquests | requests | httpx | aiohttp |
16+
|--------------------------------------------|:--------------:|:---------:|:-------------:|---------------|
17+
| `HTTP/1.1` |||||
18+
| `HTTP/2` |||[^7] ||
19+
| `HTTP/3 over QUIC` |||||
20+
| `Synchronous` |||| _N/A_[^1] |
21+
| `Asynchronous` |||||
22+
| `Thread Safe` |||[^5] | _N/A_[^1] |
23+
| `Task Safe` || _N/A_[^2] |||
24+
| `OS Trust Store` |||||
25+
| `Multiplexing` ||| _Limited_[^3] ||
26+
| `DNSSEC` |[^11] ||||
27+
| `Customizable DNS Resolution` |||||
28+
| `DNS over HTTPS` |||||
29+
| `DNS over QUIC` |||||
30+
| `DNS over TLS` |||||
31+
| `Multiple DNS Resolver` |||||
32+
| `Network Fine Tuning & Inspect` ||| _Limited_[^6] | _Limited_[^6] |
33+
| `Certificate Revocation Protection` |||||
34+
| `Session Persistence` |||||
35+
| `In-memory Certificate CA & mTLS` ||| _Limited_[^4] | _Limited_[^4] |
36+
| `SOCKS 4/5 Proxies` |||||
37+
| `HTTP/HTTPS Proxies` |||||
38+
| `TLS-in-TLS Support` |||||
39+
| `Direct HTTP/3 Negotiation` |[^9] | N/A[^8] | N/A[^8] | N/A[^8] |
40+
| `Happy Eyeballs` |||||
41+
| `Package / SLSA Signed` |||||
42+
| `HTTP/2 with prior knowledge (h2c)` |||||
43+
| `Post-Quantum Security` | _Limited_[^12] ||||
44+
| `HTTP Trailers` |||||
45+
| `Early Responses` |||||
46+
| `WebSocket over HTTP/1` ||[^14] |[^14] ||
47+
| `WebSocket over HTTP/2 and HTTP/3` |[^13] ||||
48+
| `Automatic Ping for HTTP/2+` || N/A || N/A |
49+
| `Automatic Connection Upgrade / Downgrade` | | N/A || N/A |
5050
</details>
5151

5252
<details>
@@ -67,9 +67,9 @@ _Scenario:_ Fetch a thousand requests using 10 tasks or threads, each with a hun
6767
| Client | Average Delay to Complete | Notes |
6868
|---------------|---------------------------|------------------------------|
6969
| requests core | 643 ms or ~1555 req/s | ThreadPoolExecutor. HTTP/1.1 |
70-
| httpx core | 530 ms or ~1886 req/s | Asyncio. HTTP/2 |
70+
| httpx core | 490 ms or ~2000 req/s | Asyncio. HTTP/2 |
7171
| aiohttp | 210 ms or ~4762 req/s | Asyncio. HTTP/1.1 |
72-
| niquests core | 170 ms or ~5882 req/s | Asyncio. HTTP/2 |
72+
| niquests core | 160 ms or ~6200 req/s | Asyncio. HTTP/2 |
7373

7474
Did you give up on HTTP/2 due to performance concerns? Think again! Do you realize that you can get 3 times faster with the same CPU if you ever switched to Niquests from Requests?
7575
Multiplexing and response lazyness open up a wide range of possibilities! Want to learn more about the tests? scripts? reasoning?
@@ -83,6 +83,8 @@ Take a deeper look at https://github.com/Ousret/niquests-stats
8383
>>> import niquests
8484
>>> s = niquests.Session(resolver="doh+google://", multiplexed=True)
8585
>>> r = s.get('https://pie.dev/basic-auth/user/pass', auth=('user', 'pass'))
86+
>>> r
87+
<ResponsePromise HTTP/3>
8688
>>> r.status_code
8789
200
8890
>>> r.headers['content-type']

src/niquests/__version__.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,9 @@
99
__url__: str = "https://niquests.readthedocs.io"
1010

1111
__version__: str
12-
__version__ = "3.10.0"
12+
__version__ = "3.10.1"
1313

14-
__build__: int = 0x031000
14+
__build__: int = 0x031001
1515
__author__: str = "Kenneth Reitz"
1616
__author_email__: str = "me@kennethreitz.org"
1717
__license__: str = "Apache-2.0"

src/niquests/adapters.py

+140-2
Original file line numberDiff line numberDiff line change
@@ -1232,7 +1232,42 @@ def gather(self, *responses: Response, max_fetch: int | None = None) -> None:
12321232
self._orphaned.remove(low_resp)
12331233

12341234
if low_resp is None:
1235-
low_resp = self.poolmanager.get_response()
1235+
try:
1236+
low_resp = self.poolmanager.get_response()
1237+
except (ProtocolError, OSError) as err:
1238+
raise ConnectionError(err)
1239+
1240+
except MaxRetryError as e:
1241+
if isinstance(e.reason, ConnectTimeoutError):
1242+
# TODO: Remove this in 3.0.0: see #2811
1243+
if not isinstance(e.reason, NewConnectionError):
1244+
raise ConnectTimeout(e)
1245+
1246+
if isinstance(e.reason, ResponseError):
1247+
raise RetryError(e)
1248+
1249+
if isinstance(e.reason, _ProxyError):
1250+
raise ProxyError(e)
1251+
1252+
if isinstance(e.reason, _SSLError):
1253+
# This branch is for urllib3 v1.22 and later.
1254+
raise SSLError(e)
1255+
1256+
raise ConnectionError(e)
1257+
1258+
except _ProxyError as e:
1259+
raise ProxyError(e)
1260+
1261+
except (_SSLError, _HTTPError) as e:
1262+
if isinstance(e, _SSLError):
1263+
# This branch is for urllib3 versions earlier than v1.22
1264+
raise SSLError(e)
1265+
elif isinstance(e, ReadTimeoutError):
1266+
raise ReadTimeout(e)
1267+
elif isinstance(e, _InvalidHeader):
1268+
raise InvalidHeader(e)
1269+
else:
1270+
raise
12361271

12371272
if low_resp is None:
12381273
break
@@ -1278,6 +1313,40 @@ def gather(self, *responses: Response, max_fetch: int | None = None) -> None:
12781313
)
12791314
except ValueError:
12801315
low_resp = None
1316+
except (ProtocolError, OSError) as err:
1317+
raise ConnectionError(err)
1318+
1319+
except MaxRetryError as e:
1320+
if isinstance(e.reason, ConnectTimeoutError):
1321+
# TODO: Remove this in 3.0.0: see #2811
1322+
if not isinstance(e.reason, NewConnectionError):
1323+
raise ConnectTimeout(e)
1324+
1325+
if isinstance(e.reason, ResponseError):
1326+
raise RetryError(e)
1327+
1328+
if isinstance(e.reason, _ProxyError):
1329+
raise ProxyError(e)
1330+
1331+
if isinstance(e.reason, _SSLError):
1332+
# This branch is for urllib3 v1.22 and later.
1333+
raise SSLError(e)
1334+
1335+
raise ConnectionError(e)
1336+
1337+
except _ProxyError as e:
1338+
raise ProxyError(e)
1339+
1340+
except (_SSLError, _HTTPError) as e:
1341+
if isinstance(e, _SSLError):
1342+
# This branch is for urllib3 versions earlier than v1.22
1343+
raise SSLError(e)
1344+
elif isinstance(e, ReadTimeoutError):
1345+
raise ReadTimeout(e)
1346+
elif isinstance(e, _InvalidHeader):
1347+
raise InvalidHeader(e)
1348+
else:
1349+
raise
12811350

12821351
if low_resp is None:
12831352
raise MultiplexingError(
@@ -2257,7 +2326,42 @@ async def gather(
22572326
self._orphaned.remove(low_resp)
22582327

22592328
if low_resp is None:
2260-
low_resp = await self.poolmanager.get_response()
2329+
try:
2330+
low_resp = await self.poolmanager.get_response()
2331+
except (ProtocolError, OSError) as err:
2332+
raise ConnectionError(err)
2333+
2334+
except MaxRetryError as e:
2335+
if isinstance(e.reason, ConnectTimeoutError):
2336+
# TODO: Remove this in 3.0.0: see #2811
2337+
if not isinstance(e.reason, NewConnectionError):
2338+
raise ConnectTimeout(e)
2339+
2340+
if isinstance(e.reason, ResponseError):
2341+
raise RetryError(e)
2342+
2343+
if isinstance(e.reason, _ProxyError):
2344+
raise ProxyError(e)
2345+
2346+
if isinstance(e.reason, _SSLError):
2347+
# This branch is for urllib3 v1.22 and later.
2348+
raise SSLError(e)
2349+
2350+
raise ConnectionError(e)
2351+
2352+
except _ProxyError as e:
2353+
raise ProxyError(e)
2354+
2355+
except (_SSLError, _HTTPError) as e:
2356+
if isinstance(e, _SSLError):
2357+
# This branch is for urllib3 versions earlier than v1.22
2358+
raise SSLError(e)
2359+
elif isinstance(e, ReadTimeoutError):
2360+
raise ReadTimeout(e)
2361+
elif isinstance(e, _InvalidHeader):
2362+
raise InvalidHeader(e)
2363+
else:
2364+
raise
22612365

22622366
if low_resp is None:
22632367
break
@@ -2303,6 +2407,40 @@ async def gather(
23032407
)
23042408
except ValueError:
23052409
low_resp = None
2410+
except (ProtocolError, OSError) as err:
2411+
raise ConnectionError(err)
2412+
2413+
except MaxRetryError as e:
2414+
if isinstance(e.reason, ConnectTimeoutError):
2415+
# TODO: Remove this in 3.0.0: see #2811
2416+
if not isinstance(e.reason, NewConnectionError):
2417+
raise ConnectTimeout(e)
2418+
2419+
if isinstance(e.reason, ResponseError):
2420+
raise RetryError(e)
2421+
2422+
if isinstance(e.reason, _ProxyError):
2423+
raise ProxyError(e)
2424+
2425+
if isinstance(e.reason, _SSLError):
2426+
# This branch is for urllib3 v1.22 and later.
2427+
raise SSLError(e)
2428+
2429+
raise ConnectionError(e)
2430+
2431+
except _ProxyError as e:
2432+
raise ProxyError(e)
2433+
2434+
except (_SSLError, _HTTPError) as e:
2435+
if isinstance(e, _SSLError):
2436+
# This branch is for urllib3 versions earlier than v1.22
2437+
raise SSLError(e)
2438+
elif isinstance(e, ReadTimeoutError):
2439+
raise ReadTimeout(e)
2440+
elif isinstance(e, _InvalidHeader):
2441+
raise InvalidHeader(e)
2442+
else:
2443+
raise
23062444

23072445
if low_resp is None:
23082446
raise MultiplexingError(

src/niquests/models.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -1136,7 +1136,7 @@ def __repr__(self) -> str:
11361136
or self.request.conn_info is None
11371137
or self.request.conn_info.http_version is None
11381138
):
1139-
return "<Response Dummy>"
1139+
return f"<Response Dummy [{self.status_code}]>"
11401140

11411141
# HTTP/2.0 is not preferred, cast it to HTTP/2 instead.
11421142
http_revision = self.request.conn_info.http_version.value.replace(".0", "")

0 commit comments

Comments
 (0)