Skip to content

Commit

Permalink
Add the 'proxy' parameter and deprecate 'proxies'. (#2879)
Browse files Browse the repository at this point in the history 
* Add the proxy parameter and deprecate proxies

* Make the Client.proxy and HTTPTransport.proxy types the same

* Update httpx/_transports/default.py

Co-authored-by: T-256 <132141463+T-256@users.noreply.github.com>

* Update httpx/_transports/default.py

Co-authored-by: T-256 <132141463+T-256@users.noreply.github.com>

* Drop unneeded noqa

* Changelog

* update documentation

* Allow None in mounts

* typos

* Update httpx/_types.py

* Changes proxies to proxy in CLI app

* Add proxy to request function

* Update CHANGELOG.md

Co-authored-by: Tom Christie <tom@tomchristie.com>

* Update docs/troubleshooting.md

Co-authored-by: Tom Christie <tom@tomchristie.com>

* Update docs/troubleshooting.md

Co-authored-by: Tom Christie <tom@tomchristie.com>

* Lint

---------

Co-authored-by: Tom Christie <tom@tomchristie.com>
Co-authored-by: T-256 <132141463+T-256@users.noreply.github.com>
  • Loading branch information
@karpetrosyan @tomchristie @T-256
3 people authored Dec 11, 2023
1 parent b471f01 commit f8981f3
Show file tree
Hide file tree
Showing 11 changed files with 281 additions and 170 deletions.
4 changes: 4 additions & 0 deletions CHANGELOG.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## Unreleased

### Changed

* The `proxies` argument is now deprecated. You should use the `proxy` argument instead, or use `mounts=` for more complex configurations. (#2879)

### Fixed

* Allow URLs where username or password contains unescaped '@'. (#2986)
Expand Down
264 changes: 132 additions & 132 deletions docs/advanced.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -504,7 +504,7 @@ The `NetRCAuth()` class uses [the `netrc.netrc()` function from the Python stand

## HTTP Proxying

HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxies` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxies=...)`.
HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_server#Web_proxy_servers) via the `proxy` parameter to be passed on client initialization or top-level API functions like `httpx.get(..., proxy=...)`.

<div align="center">
<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/2/27/Open_proxy_h2g2bob.svg/480px-Open_proxy_h2g2bob.svg.png"/>
Expand All @@ -516,19 +516,19 @@ HTTPX supports setting up [HTTP proxies](https://en.wikipedia.org/wiki/Proxy_ser
To route all traffic (HTTP and HTTPS) to a proxy located at `http://localhost:8030`, pass the proxy URL to the client...

```python
with httpx.Client(proxies="http://localhost:8030") as client:
with httpx.Client(proxy="http://localhost:8030") as client:
...
```

For more advanced use cases, pass a proxies `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs:
For more advanced use cases, pass a mounts `dict`. For example, to route HTTP and HTTPS requests to 2 different proxies, respectively located at `http://localhost:8030`, and `http://localhost:8031`, pass a `dict` of proxy URLs:

```python
proxies = {
"http://": "http://localhost:8030",
"https://": "http://localhost:8031",
proxy_mounts = {
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}

with httpx.Client(proxies=proxies) as client:
with httpx.Client(mounts=proxy_mounts) as client:
...
```

Expand All @@ -546,132 +546,10 @@ For detailed information about proxy routing, see the [Routing](#routing) sectio
Proxy credentials can be passed as the `userinfo` section of the proxy URL. For example:

```python
proxies = {
"http://": "http://username:password@localhost:8030",
# ...
}
```

### Routing

HTTPX provides fine-grained controls for deciding which requests should go through a proxy, and which shouldn't. This process is known as proxy routing.

The `proxies` dictionary maps URL patterns ("proxy keys") to proxy URLs. HTTPX matches requested URLs against proxy keys to decide which proxy should be used, if any. Matching is done from most specific proxy keys (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

HTTPX supports routing proxies based on **scheme**, **domain**, **port**, or a combination of these.

#### Wildcard routing

Route everything through a proxy...

```python
proxies = {
"all://": "http://localhost:8030",
}
```

#### Scheme routing

Route HTTP requests through one proxy, and HTTPS requests through another...

```python
proxies = {
"http://": "http://localhost:8030",
"https://": "http://localhost:8031",
}
```

#### Domain routing

Proxy all requests on domain "example.com", let other requests pass through...

```python
proxies = {
"all://example.com": "http://localhost:8030",
}
```

Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through...

```python
proxies = {
"http://example.com": "http://localhost:8030",
}
```

Proxy all requests to "example.com" and its subdomains, let other requests pass through...

```python
proxies = {
"all://*example.com": "http://localhost:8030",
}
```

Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through...

```python
proxies = {
"all://*.example.com": "http://localhost:8030",
}
```

#### Port routing

Proxy HTTPS requests on port 1234 to "example.com"...

```python
proxies = {
"https://example.com:1234": "http://localhost:8030",
}
```

Proxy all requests on port 1234...

```python
proxies = {
"all://*:1234": "http://localhost:8030",
}
```

#### No-proxy support

It is also possible to define requests that _shouldn't_ be routed through proxies.

To do so, pass `None` as the proxy URL. For example...

```python
proxies = {
# Route requests through a proxy by default...
"all://": "http://localhost:8031",
# Except those for "example.com".
"all://example.com": None,
}
```

#### Complex configuration example

You can combine the routing features outlined above to build complex proxy routing configurations. For example...

```python
proxies = {
# Route all traffic through a proxy by default...
"all://": "http://localhost:8030",
# But don't use proxies for HTTPS requests to "domain.io"...
"https://domain.io": None,
# And use another proxy for requests to "example.com" and its subdomains...
"all://*example.com": "http://localhost:8031",
# And yet another proxy if HTTP is used,
# and the "internal" subdomain on port 5550 is requested...
"http://internal.example.com:5550": "http://localhost:8032",
}
with httpx.Client(proxy="http://username:password@localhost:8030") as client:
...
```

#### Environment variables

HTTP proxying can also be configured through environment variables, although with less fine-grained control.

See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information.

### Proxy mechanisms

!!! note
Expand Down Expand Up @@ -707,7 +585,7 @@ $ pip install httpx[socks]
You can now configure a client to make requests via a proxy using the SOCKS protocol:

```python
httpx.Client(proxies='socks5://user:pass@host:port')
httpx.Client(proxy='socks5://user:pass@host:port')
```

## Timeout Configuration
Expand Down Expand Up @@ -1294,3 +1172,125 @@ Adding support for custom schemes:
mounts = {"file://": FileSystemTransport()}
client = httpx.Client(mounts=mounts)
```

### Routing

HTTPX provides a powerful mechanism for routing requests, allowing you to write complex rules that specify which transport should be used for each request.

The `mounts` dictionary maps URL patterns to HTTP transports. HTTPX matches requested URLs against URL patterns to decide which transport should be used, if any. Matching is done from most specific URL patterns (e.g. `https://<domain>:<port>`) to least specific ones (e.g. `https://`).

HTTPX supports routing requests based on **scheme**, **domain**, **port**, or a combination of these.

#### Wildcard routing

Route everything through a transport...

```python
mounts = {
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### Scheme routing

Route HTTP requests through one transport, and HTTPS requests through another...

```python
mounts = {
"http://": httpx.HTTPTransport(proxy="http://localhost:8030"),
"https://": httpx.HTTPTransport(proxy="http://localhost:8031"),
}
```

#### Domain routing

Proxy all requests on domain "example.com", let other requests pass through...

```python
mounts = {
"all://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy HTTP requests on domain "example.com", let HTTPS and other requests pass through...

```python
mounts = {
"http://example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests to "example.com" and its subdomains, let other requests pass through...

```python
mounts = {
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests to strict subdomains of "example.com", let "example.com" and other requests pass through...

```python
mounts = {
"all://*.example.com": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### Port routing

Proxy HTTPS requests on port 1234 to "example.com"...

```python
mounts = {
"https://example.com:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

Proxy all requests on port 1234...

```python
mounts = {
"all://*:1234": httpx.HTTPTransport(proxy="http://localhost:8030"),
}
```

#### No-proxy support

It is also possible to define requests that _shouldn't_ be routed through the transport.

To do so, pass `None` as the proxy URL. For example...

```python
mounts = {
# Route requests through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8031"),
# Except those for "example.com".
"all://example.com": None,
}
```

#### Complex configuration example

You can combine the routing features outlined above to build complex proxy routing configurations. For example...

```python
mounts = {
# Route all traffic through a proxy by default...
"all://": httpx.HTTPTransport(proxy="http://localhost:8030"),
# But don't use proxies for HTTPS requests to "domain.io"...
"https://domain.io": None,
# And use another proxy for requests to "example.com" and its subdomains...
"all://*example.com": httpx.HTTPTransport(proxy="http://localhost:8031"),
# And yet another proxy if HTTP is used,
# and the "internal" subdomain on port 5550 is requested...
"http://internal.example.com:5550": httpx.HTTPTransport(proxy="http://localhost:8032"),
}
```

#### Environment variables

There are also environment variables that can be used to control the dictionary of the client mounts.
They can be used to configure HTTP proxying for clients.

See documentation on [`HTTP_PROXY`, `HTTPS_PROXY`, `ALL_PROXY`](environment_variables.md#http_proxy-https_proxy-all_proxy) for more information.

12 changes: 8 additions & 4 deletions docs/compatibility.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -157,13 +157,17 @@ httpx.get('https://www.example.com', timeout=None)

## Proxy keys

When using `httpx.Client(proxies={...})` to map to a selection of different proxies, we use full URL schemes, such as `proxies={"http://": ..., "https://": ...}`.
HTTPX uses the mounts argument for HTTP proxying and transport routing.
It can do much more than proxies and allows you to configure more than just the proxy route.
For more detailed documentation, see [Mounting Transports](advanced.md#mounting-transports).

When using `httpx.Client(mounts={...})` to map to a selection of different transports, we use full URL schemes, such as `mounts={"http://": ..., "https://": ...}`.

This is different to the `requests` usage of `proxies={"http": ..., "https": ...}`.

This change is for better consistency with more complex mappings, that might also include domain names, such as `proxies={"all://": ..., "all://www.example.com": None}` which maps all requests onto a proxy, except for requests to "www.example.com" which have an explicit exclusion.
This change is for better consistency with more complex mappings, that might also include domain names, such as `mounts={"all://": ..., httpx.HTTPTransport(proxy="all://www.example.com": None})` which maps all requests onto a proxy, except for requests to "www.example.com" which have an explicit exclusion.

Also note that `requests.Session.request(...)` allows a `proxies=...` parameter, whereas `httpx.Client.request(...)` does not.
Also note that `requests.Session.request(...)` allows a `proxies=...` parameter, whereas `httpx.Client.request(...)` does not allow `mounts=...`.

## SSL configuration

Expand Down Expand Up @@ -195,7 +199,7 @@ We don't support `response.is_ok` since the naming is ambiguous there, and might

There is no notion of [prepared requests](https://requests.readthedocs.io/en/stable/user/advanced/#prepared-requests) in HTTPX. If you need to customize request instantiation, see [Request instances](advanced.md#request-instances).

Besides, `httpx.Request()` does not support the `auth`, `timeout`, `follow_redirects`, `proxies`, `verify` and `cert` parameters. However these are available in `httpx.request`, `httpx.get`, `httpx.post` etc., as well as on [`Client` instances](advanced.md#client-instances).
Besides, `httpx.Request()` does not support the `auth`, `timeout`, `follow_redirects`, `mounts`, `verify` and `cert` parameters. However these are available in `httpx.request`, `httpx.get`, `httpx.post` etc., as well as on [`Client` instances](advanced.md#client-instances).

## Mocking

Expand Down
4 changes: 1 addition & 3 deletions docs/contributing.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -213,9 +213,7 @@ this is where our previously generated `client.pem` comes in:
```
import httpx
proxies = {"all://": "http://127.0.0.1:8080/"}
with httpx.Client(proxies=proxies, verify="/path/to/client.pem") as client:
with httpx.Client(proxy="http://127.0.0.1:8080/", verify="/path/to/client.pem") as client:
response = client.get("https://example.org")
print(response.status_code) # should print 200
```
Expand Down
16 changes: 9 additions & 7 deletions docs/troubleshooting.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,9 @@ httpx.ProxyError: _ssl.c:1091: The handshake operation timed out
**Resolution**: it is likely that you've set up your proxies like this...

```python
proxies = {
"http://": "http://myproxy.org",
"https://": "https://myproxy.org",
mounts = {
"http://": httpx.HTTPTransport(proxy="http://myproxy.org"),
"https://": httpx.HTTPTransport(proxy="https://myproxy.org"),
}
```

Expand All @@ -32,16 +32,18 @@ But if you get the error above, it is likely that your proxy doesn't support con
Change the scheme of your HTTPS proxy to `http://...` instead of `https://...`:

```python
proxies = {
"http://": "http://myproxy.org",
"https://": "http://myproxy.org",
mounts = {
"http://": httpx.HTTPTransport(proxy="http://myproxy.org"),
"https://": httpx.HTTPTransport(proxy="http://myproxy.org"),
}
```

This can be simplified to:

```python
proxies = "http://myproxy.org"
proxy = "http://myproxy.org"
with httpx.Client(proxy=proxy) as client:
...
```

For more information, see [Proxies: FORWARD vs TUNNEL](advanced.md#forward-vs-tunnel).
Expand Down
Loading

0 comments on commit f8981f3

Please sign in to comment.