Sync new Zyte API features (#218)

Author: Gallaecio

.github/workflows/test.yml

@@ -29,6 +29,8 @@ jobs:
           toxenv: pinned-scrapy-2x5
         - python-version: '3.9'
           toxenv: pinned-scrapy-2x6
+        - python-version: '3.9'
+          toxenv: pinned-scrapy-2x7
         - python-version: '3.10'
         - python-version: '3.11'
         - python-version: '3.12'

CHANGES.rst

@@ -1,6 +1,51 @@
 Changes
 =======
 
+Unreleased
+----------
+
+* Added :ref:`automatic mapping <automap>` support for new Zyte API request
+  fields:
+  :http:`request:customAttributes`,
+  :http:`request:customAttributesOptions`,
+  :http:`request:ipType`,
+  :http:`request:followRedirect`,
+  :http:`request:forumThread`,
+  :http:`request:forumThreadOptions`,
+  :http:`request:jobPostingNavigation`,
+  :http:`request:jobPostingNavigationOptions`,
+  :http:`request:networkCapture`,
+  :http:`request:serp`,
+  :http:`request:serpOptions`,
+  :http:`request:session`,
+  :http:`request:tags`.
+
+  * You will now be warned when using their default values unnecessarily.
+
+  * By default, the following fields no longer affect request fingerprinting
+    (i.e. 2 request identical except for the value of that field are now
+    considered duplicate requests): :http:`request:ipType`,
+    :http:`request:session`.
+
+  * When enabling :http:`request:serp`, :http:`request:httpResponseBody` and
+    :http:`request:httpResponseHeaders` will no longer be enabled by default,
+    and header mapping is disabled.
+
+* Session pool IDs, of server-managed sessions (:http:`request:sessionContext`)
+  or :ref:`set through the session management API <session-pools>`, now affect
+  request fingerprinting: 2 requests identical except for their session pool ID
+  are *not* considered duplicate requests any longer.
+
+* When it is not clear whether a request will use browser rendering or not,
+  e.g. an :ref:`automatic extraction request <zapi-extract>` without an
+  :http:`extractFrom <request:productOptions.extractFrom>` value, the URL
+  fragment is now taken into account for request fingerprinting, i.e.
+  ``https://example.com#a`` and ``https://example.com#b`` are *not* considered
+  duplicate requests anymore in those scenarios.
+
+* Fixes ``"auto"`` being considered the default value of :http:`request:device`
+  instead of ``"desktop"``.
+
 0.27.0 (2025-02-04)
 -------------------
 

docs/reference/fingerprint-params.rst

@@ -7,49 +7,68 @@ Request fingerprinting parameters
 The request fingerprinter class of scrapy-zyte-api generates request
 fingerprints for Zyte API requests based on the following Zyte API parameters:
 
--   :http:`request:url` (:func:`canonicalized <w3lib.url.canonicalize_url>`)
+-   :http:`request:url` (:func:`canonicalized <w3lib.url.canonicalize_url>`).
 
     For URLs that include a URL fragment, like ``https://example.com#foo``, URL
-    canonicalization keeps the URL fragment if :http:`request:browserHtml` or
-    :http:`request:screenshot` are enabled, or if extractFrom_ is set to
-    ``browserHtml``.
-
-    .. _extractFrom: https://docs.zyte.com/zyte-api/usage/extract.html#extraction-source
+    canonicalization keeps the URL fragment if the request *may* be a browser
+    request.
 
 -   Request attribute parameters (:http:`request:httpRequestBody`,
     :http:`request:httpRequestText`, :http:`request:httpRequestMethod`), except
-    headers
+    headers.
 
     Equivalent :http:`request:httpRequestBody` and
     :http:`request:httpRequestText` values generate the same signature.
 
 -   Output parameters (:http:`request:browserHtml`,
     :http:`request:httpResponseBody`, :http:`request:httpResponseHeaders`,
-    :http:`request:responseCookies`, :http:`request:screenshot`, and
+    :http:`request:responseCookies`, :http:`request:screenshot`,
     :ref:`automatic extraction outputs <zapi-extract-fields>` like
-    :http:`request:product`)
+    :http:`request:product`, and :http:`request:customAttributes`).
+
+    Same for :http:`request:networkCapture`, although it is not a proper output
+    parameter (it needs to be combined with another browser rendering parameter
+    to work).
 
 -   Rendering option parameters (:http:`request:actions`,
     :http:`request:device`, :http:`request:javascript`,
     :http:`request:screenshotOptions`, :http:`request:viewport`, and automatic
-    extraction options like :http:`request:productOptions`)
+    extraction options like :http:`request:productOptions` or
+    :http:`request:customAttributesOptions`).
+
+-   :http:`request:geolocation`.
+
+-   :http:`request:sessionContext`.
+
+    When using the :ref:`session management API <session>`, :ref:`session pool
+    IDs <session-pools>` are treated the same as
+    :http:`request:sessionContext`.
 
--   :http:`request:geolocation`
+-   :http:`request:followRedirect`.
 
--   :http:`request:echoData`
+-   :http:`request:echoData`.
+
+-   :http:`request:tags`.
 
 The following Zyte API parameters are *not* taken into account for request
-fingerprinting:
+fingerprinting by default:
 
 -   Request header parameters (:http:`request:customHttpRequestHeaders`,
-    :http:`request:requestHeaders`)
+    :http:`request:requestHeaders`).
 
 -   Request cookie parameters (:http:`request:cookieManagement`,
-    :http:`request:requestCookies`)
+    :http:`request:requestCookies`).
+
+-   :http:`request:sessionContextParameters`.
+
+    When using the :ref:`session management API <session>`, :ref:`session
+    initialization parameters <session-init>` are treated the same as
+    :http:`request:sessionContextParameters`.
+
+-   :http:`request:session.id`.
 
--   Session handling parameters (:http:`request:sessionContext`,
-    :http:`request:sessionContextParameters`)
+-   :http:`request:ipType`.
 
--   :http:`request:jobId`
+-   :http:`request:jobId`.
 
--   Experimental parameters (:http:`experimental.* <request:experimental>`)
+-   Experimental parameters (:http:`experimental.* <request:experimental>`).

docs/usage/automap.rst

@@ -45,7 +45,7 @@ Changing parameters
 
 You may set :reqmeta:`zyte_api_automap` in :attr:`Request.meta
 <scrapy.http.Request.meta>` to a :class:`dict` of Zyte API parameters to add,
-modify, or remove (by setting to ``False``) automatic request parameters. This
+modify, or remove (by setting to ``None``) automatic request parameters. This
 also works in :ref:`transparent mode <transparent>`.
 
 Enabling :http:`request:browserHtml`, :http:`request:screenshot`, or an

docs/usage/default.rst

@@ -17,6 +17,9 @@ in all requests:
 -   :setting:`ZYTE_API_DEFAULT_PARAMS`, for :ref:`manual request parameters
     <manual>`.
 
+-   :setting:`ZYTE_API_PROVIDER_PARAMS`, for :ref:`dependency injection
+    <scrapy-poet>`.
+
 For example, if you set :setting:`ZYTE_API_DEFAULT_PARAMS` to
 ``{"geolocation": "US"}`` and :reqmeta:`zyte_api` to ``{"browserHtml": True}``,
 ``{"url: "…", "geolocation": "US", "browserHtml": True}`` is sent to Zyte API.

docs/usage/fingerprint.rst

@@ -17,10 +17,10 @@ Use :setting:`ZYTE_API_FALLBACK_REQUEST_FINGERPRINTER_CLASS` to define a custom
 request fingerprinting for requests that do not go through Zyte API.
 
 
-Request fingerprinting before Scrapy 2.7
-----------------------------------------
+Request fingerprinting below Scrapy 2.7
+---------------------------------------
 
-If you have a Scrapy version older than Scrapy 2.7, Zyte API parameters are not
+If you have a Scrapy version lower than Scrapy 2.7, Zyte API parameters are not
 taken into account for request fingerprinting. This can cause some Scrapy
 components, like the filter of duplicate requests or the HTTP cache extension,
 to interpret 2 different requests as being the same.

docs/usage/session.rst

@@ -71,7 +71,7 @@ By default, scrapy-zyte-api will maintain up to 8 sessions per domain, each
 initialized with a :ref:`browser request <zapi-browser>` targeting the URL
 of the first request that will use the session. Sessions are automatically
 rotated among requests, and refreshed as they expire or get banned. You can
-customize most of this logic though request metadata, settings and
+customize most of this logic through request metadata, settings and
 :ref:`session config overrides <session-configs>`.
 
 For session management to work as expected, your
@@ -269,6 +269,11 @@ sessions:
     initialization request <session-init>` is triggered to replace that
     session in the session pool.
 
+The session pool assigned to a request affects the :ref:`fingerprint
+<fingerprint>` of the request. 2 requests with a different session pool ID are
+considered different requests, i.e. not duplicate requests, even if they are
+otherwise identical.
+
 
 .. _optimize-sessions: