Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make async dependencies optional. #2858

Open
wants to merge 15 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 11 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,12 @@ Install with pip:
$ pip install httpx
```

Or, to include the optional HTTP/2 support, use:
There are also a number of optional dependancies.

For example to include asyncio and HTTP/2 support, use:

```shell
$ pip install httpx[http2]
$ pip install 'httpx[asyncio,http2]'
```

HTTPX requires Python 3.8+.
Expand Down Expand Up @@ -129,15 +131,16 @@ The HTTPX project relies on these excellent libraries:
* `h11` - HTTP/1.1 support.
* `certifi` - SSL certificates.
* `idna` - Internationalized domain name support.
* `sniffio` - Async library autodetection.

As well as these optional installs:

* `h2` - HTTP/2 support. *(Optional, with `httpx[http2]`)*
* `socksio` - SOCKS proxy support. *(Optional, with `httpx[socks]`)*
* `rich` - Rich terminal support. *(Optional, with `httpx[cli]`)*
* `click` - Command line client support. *(Optional, with `httpx[cli]`)*
* `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx[brotli]`)*
* `anyio`, `sniffio` - Async support for `asyncio`. *(Optional, with `httpx['asyncio']`)*
* `trio`, `sniffio` - Async support for `trio`. *(Optional, with `httpx['trio']`)*
tomchristie marked this conversation as resolved.
Show resolved Hide resolved
* `h2` - HTTP/2 support. *(Optional, with `httpx['http2']`)*
* `socksio` - SOCKS proxy support. *(Optional, with `httpx['socks']`)*
tomchristie marked this conversation as resolved.
Show resolved Hide resolved
* `rich` - Rich terminal support. *(Optional, with `httpx['cli']`)*
* `click` - Command line client support. *(Optional, with `httpx['cli']`)*
tomchristie marked this conversation as resolved.
Show resolved Hide resolved
* `brotli` or `brotlicffi` - Decoding for "brotli" compressed responses. *(Optional, with `httpx['brotli']`)*

A huge amount of credit is due to `requests` for the API layout that
much of this work follows, as well as to `urllib3` for plenty of design
Expand Down
18 changes: 18 additions & 0 deletions docs/async.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,24 @@ long-lived network connections such as WebSockets.
If you're working with an async web framework then you'll also want to use an
async client for sending outgoing HTTP requests.

## Enabling Async support

To enable async support you'll need to install some additional dependencies:

If you're using Python's [standard `asyncio` support](https://docs.python.org/3/library/asyncio.html) then:

```shell
$ pip install httpx['asyncio']
tomchristie marked this conversation as resolved.
Show resolved Hide resolved
```

Or, if you're working with the [`trio` third party package](https://trio.readthedocs.io/en/stable/):

```shell
$ pip install httpx['trio']
```

We highly recommend `trio` for async support. The `trio` project [pioneered the principles of structured concurrency](https://en.wikipedia.org/wiki/Structured_concurrency), and has a more carefully constrained API against which to work from.

## Making Async requests

To make asynchronous requests, you'll need an `AsyncClient`.
Expand Down
2 changes: 1 addition & 1 deletion docs/http2.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ trying out our HTTP/2 support. You can do so by first making sure to install
the optional HTTP/2 dependencies...

```shell
$ pip install httpx[http2]
$ pip install httpx['http2']
tomchristie marked this conversation as resolved.
Show resolved Hide resolved
```

And then instantiating a client with HTTP/2 support enabled:
Expand Down
4 changes: 2 additions & 2 deletions httpx/_transports/asgi.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,5 @@
import typing

import sniffio

from .._models import Request, Response
from .._types import AsyncByteStream
from .base import AsyncBaseTransport
Expand All @@ -25,6 +23,8 @@


def create_event() -> "Event":
import sniffio

if sniffio.current_async_library() == "trio":
import trio

Expand Down
9 changes: 8 additions & 1 deletion httpx/_transports/default.py
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,7 @@
def map_httpcore_exceptions() -> typing.Iterator[None]:
try:
yield
except Exception as exc: # noqa: PIE-786
except Exception as exc:
mapped_exc = None

for from_exc, to_exc in HTTPCORE_EXC_MAP.items():
Expand Down Expand Up @@ -269,6 +269,13 @@ def __init__(
retries: int = 0,
socket_options: typing.Optional[typing.Iterable[SOCKET_OPTION]] = None,
) -> None:
try:
import sniffio # noqa: F401
except ImportError: # pragma: nocover
raise RuntimeError(
"Using httpx in async mode, but neither httpx['asyncio'] or asyncio['trio'] is installed."
)
tomchristie marked this conversation as resolved.
Show resolved Hide resolved

ssl_context = create_ssl_context(verify=verify, cert=cert, trust_env=trust_env)

if proxy is None:
Expand Down
21 changes: 2 additions & 19 deletions httpx/_utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,6 @@
from pathlib import Path
from urllib.request import getproxies

import sniffio

from ._types import PrimitiveData

if typing.TYPE_CHECKING: # pragma: no cover
Expand Down Expand Up @@ -322,33 +320,18 @@ def peek_filelike_length(stream: typing.Any) -> typing.Optional[int]:


class Timer:
async def _get_time(self) -> float:
library = sniffio.current_async_library()
if library == "trio":
import trio

return trio.current_time()
elif library == "curio": # pragma: no cover
import curio

return typing.cast(float, await curio.clock())

import asyncio

return asyncio.get_event_loop().time()

def sync_start(self) -> None:
self.started = time.perf_counter()

async def async_start(self) -> None:
self.started = await self._get_time()
self.started = time.perf_counter()

def sync_elapsed(self) -> float:
now = time.perf_counter()
return now - self.started

async def async_elapsed(self) -> float:
now = await self._get_time()
now = time.perf_counter()
return now - self.started


Expand Down
13 changes: 9 additions & 4 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -29,9 +29,8 @@ classifiers = [
]
dependencies = [
"certifi",
"httpcore>=0.18.0,<0.19.0",
"httpcore>=1.0.0,<2.0.0",
"idna",
"sniffio",
]
dynamic = ["readme", "version"]

Expand All @@ -46,10 +45,16 @@ cli = [
"rich>=10,<14",
]
http2 = [
"h2>=3,<5",
"httpcore[http2]",
]
socks = [
"socksio==1.*",
"httpcore[socks]",
]
asyncio = [
"httpcore[asyncio]"
]
trio = [
"httpcore[trio]"
]
Comment on lines 47 to 58
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is interesting... currently we're including the httpcore dependencies explicitly.

Eg. pip install httpx[socks] uses the dependency socksio==1.* instead of the dependency httpcore[socks].

The second style makes more sense, because that's what we're actually trying to effect... but I'm a bit hesitant that some Python install tooling might incorrectly fail to install httpcore[socks] as an optional dependency when the httpcore dependancy is already included. Is my hesitancy well founded here, or are we okay to use this new style throughout?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this should always work but you're right that it's tool dependent and the Python specification around this does not provide guarantees.


[project.scripts]
Expand Down
3 changes: 1 addition & 2 deletions requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# On the other hand, we're not pinning package dependencies, because our tests
# needs to pass with the latest version of the packages.
# Reference: https://github.com/encode/httpx/pull/1721#discussion_r661241588
-e .[brotli,cli,http2,socks]
-e .[asyncio,trio,brotli,cli,http2,socks]

# Optional charset auto-detection
# Used in our test cases
Expand All @@ -26,7 +26,6 @@ mypy==1.5.1
types-certifi==2021.10.8.2
pytest==7.4.2
ruff==0.0.291
trio==0.22.2
trio-typing==0.8.0
trustme==1.1.0
uvicorn==0.22.0