Skip to content

Commit

Permalink
Bump curl-impersonate to 0.9.3, close #500, close #478, close #484
Browse files Browse the repository at this point in the history
  • Loading branch information
lexiforest committed Feb 15, 2025
1 parent 6993226 commit f6f9006
Show file tree
Hide file tree
Showing 8 changed files with 72 additions and 18 deletions.
2 changes: 1 addition & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
SHELL := bash

# this is the upstream libcurl-impersonate version
VERSION := 0.9.1
VERSION := 0.9.3
CURL_VERSION := curl-8_7_1

$(CURL_VERSION):
Expand Down
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,7 @@ to specify your own customized fingerprints. See the [docs on impersonatation](h
- chrome123 <sup>[3]</sup>
- chrome124 <sup>[3]</sup>
- chrome131 <sup>[4]</sup>
- chrome133a <sup>[5][6]</sup>
- chrome99_android
- chrome131_android <sup>[4]</sup>
- edge99
Expand All @@ -182,6 +183,7 @@ Notes:
3. Added in version `0.7.0`.
4. Added in version `0.8.0`.
5. Added in version `0.9.0`.
6. The version postfix `-a`(e.g. `chrome133a`) means that this is an alternative version, i.e. the fingerprint has not been officially updated by browser, but has been observed because of A/B testing.

### asyncio

Expand Down
45 changes: 43 additions & 2 deletions docs/advanced.rst
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
Advanced Usage
==============

low-level curl API
Low-level curl API
---------

Alternatively, you can use the low-level curl-like API:
Expand All @@ -24,7 +24,7 @@ Alternatively, you can use the low-level curl-like API:
print(body.decode())
scrapy integrations
Scrapy integrations
------

If you are using scrapy, check out these middlewares:
Expand All @@ -44,3 +44,44 @@ Just set ``thread`` to eventlet or gevent.
s = requests.Session(thread="eventlet")
s.get(url)
As a urllib3/requests adapter
------

You can also use curl-cffi as a requests adapter via `curl-adapter <https://github.com/el1s7/curl-adapter>`_.
In this way, you get the full functionality of requests.

.. code-block:: python
import requests
from curl_adapter import CurlCffiAdapter
session = requests.Session()
session.mount("http://", CurlCffiAdapter())
session.mount("https://", CurlCffiAdapter())
# just use requests session like you normally would
session.get("https://example.com")
As a httpx transport
------

You can also use curl-cffi as a httpx transport via `httpx-curl-cffi <https://github.com/vgavro/httpx-curl-cffi>`_.
With this, you get the full functionality of httpx.

.. code-block:: python
from httpx import Client, AsyncClient
from httpx_curl_cffi import CurlTransport, AsyncCurlTransport, CurlOpt
client = Client(transport=CurlTransport(impersonate="chrome", default_headers=True))
client.get("https://tools.scrapfly.io/api/fp/ja3")
async_client = AsyncClient(transport=AsyncCurlTransport(
impersonate="chrome",
default_headers=True,
# required for parallel requests, see curl_cffi issues below
curl_options={CurlOpt.FRESH_CONNECT: True}
))
8 changes: 8 additions & 0 deletions docs/changelog.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,14 @@ Change Log

Please see the `GitHub Releases <https://github.com/lexiforest/curl_cffi/releases>`_ page for details.

v0.9
----

- v0.9.0
- Brought back Windows support
- Added support for Firefox
- Added support for Chrome 133a

v0.8
----

Expand Down
4 changes: 4 additions & 0 deletions docs/impersonate.rst
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ to specify your own customized fingerprints. See below for details.
- chrome123 :sup:`3`
- chrome124 :sup:`3`
- chrome131 :sup:`4`
- chrome133a :sup:`5` :sup:`6`
- chrome99_android
- chrome131_android :sup:`4`
- edge99
Expand All @@ -36,13 +37,16 @@ to specify your own customized fingerprints. See below for details.
- safari17_2_ios :sup:`1`
- safari18_0 :sup:`4`
- safari18_0_ios :sup:`4`
- firefox133 :sup:`5`

Notes:

1. Added in version `0.6.0`.
2. Fixed in version `0.6.0`, previous http2 fingerprints were `not correct <https://github.com/lwthiker/curl-impersonate/issues/215>`_.
3. Added in version `0.7.0`.
4. Added in version `0.8.0`.
5. Added in version `0.9.0`.
6. The version postfix `-a`(e.g. `chrome133a`) means that this is an alternative version, i.e. the fingerprint has not been officially updated by browser, but has been observed because of A/B testing.

Which version to use?
---------------------
Expand Down
24 changes: 10 additions & 14 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -35,21 +35,17 @@ website for no obvious reason, you can give this package a try.

------

.. image:: https://raw.githubusercontent.com/lexiforest/curl_cffi/main/assets/scrapfly.png
.. image:: https://raw.githubusercontent.com/lexiforest/curl_cffi/main/assets/proxycurl.png
:width: 300
:alt: Scrapfly
:target: https://scrapfly.io/?utm_source=github&utm_medium=sponsoring&utm_campaign=curl_cffi

`Scrapfly <https://scrapfly.io/?utm_source=github&utm_medium=sponsoring&utm_campaign=curl_cffi>`_
is an enterprise-grade solution providing Web Scraping API that aims to simplify the
scraping process by managing everything: real browser rendering, rotating proxies, and
fingerprints (TLS, HTTP, browser) to bypass all major anti-bots. Scrapfly also unlocks the
observability by providing an analytical dashboard and measuring the success rate/block
rate in details.

Scrapfly is a good solution if you are looking for a cloud-managed solution for ``curl_cffi``.
If you are managing TLS/HTTP fingerprint by yourself with ``curl_cffi``, they also maintain
a `curl to python converter <https://scrapfly.io/web-scraping-tools/curl-python/curl_cffi>`_ .
:alt: ProxyCurl
:target: https://nubela.co/proxycurl/?utm_campaign=influencer_marketing&utm_source=github&utm_medium=social&utm_term=-&utm_content=lexiforest-curl_cffi

Scrape public LinkedIn profile data at scale with `Proxycurl APIs <https://nubela.co/proxycurl/?utm_campaign=influencer_marketing&utm_source=github&utm_medium=social&utm_term=-&utm_content=lexiforest-curl_cffi>`_. Built for developers, by developers.

- GDPR, CCPA, SOC2 compliant
- High rate limit (300 requests/min), Fast (APIs respond in ~2s), High accuracy
- Fresh data - 88% of data is scraped real-time, other 12% is <29 days
- Tons of data points returned per profile

------

Expand Down
3 changes: 3 additions & 0 deletions docs/vs-requests.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,6 @@ Here are a list of known incompatibilities:
- retries are not supported yet, tracked in [#24](https://github.com/lexiforest/curl_cffi/issues/24)
- redirect history are not supported, tracked in [#82](https://github.com/lexiforest/curl_cffi/issues/82)
- empty-domains cookies may lost during redirects, tracked in [#55](https://github.com/lexiforest/curl_cffi/issues/55)

Alternatively, you can also use curl-cffi as a requests adapter via `curl-adapter <https://github.com/el1s7/curl-adapter>`_.
In this way, you get the full functionality of requests.
2 changes: 1 addition & 1 deletion scripts/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from cffi import FFI

# this is the upstream libcurl-impersonate version
__version__ = "0.9.1"
__version__ = "0.9.3"


def detect_arch():
Expand Down

0 comments on commit f6f9006

Please sign in to comment.