Skip to content

PEP 803: Updates based on packaging discussion #4718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

PEP 803: Updates based on packaging discussion #4718

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 133 additions & 21 deletions peps/pep-0803.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,12 +63,13 @@ to those builds.
To build against the Stable ABI, the extension must use a *Limited API*,
that is, only a subset of the functions, structures, etc. that CPython
exposes.
The Limited API is versioned, and building against Limited API 3.X
yields an extension that is ABI-compatible with CPython 3.X and *any* later
version (though bugs in CPython sometimes cause incompatibilities in practice).
Also, the Limited API is not “stable”: newer versions may remove API that
were a part of older versions.
Both the Limited API and the Stable ABI are versioned, and building against
Stable ABI 3.X requires using only Limited API 3.X, and yields an extension
that is ABI-compatible with CPython 3.X and *any* later version
(though bugs in CPython sometimes cause incompatibilities in practice).

The Limited API is not “stable”: newer versions may remove API that
were a part of older versions.
This PEP proposes the most significant such removal to date.


Expand All @@ -88,6 +89,7 @@ No backwards compatibility now

However, we won't block the possibility of extending compatibility to
CPython 3.14 and below.
See a :ref:`rejected idea <pep803-no-shim>` for how this could work.

API changes are OK
The new Limited API may require extension authors to make significant
Expand Down Expand Up @@ -172,9 +174,12 @@ New Export Hook (PEP 793)
-------------------------

Implementation of this PEP requires :pep:`793` (``PyModExport``):
A new entry point for C extension modules) to be
accepted, providing a new “export hook” for defining extension modules.
Using the new hook will become mandatory in Limited API 3.15.
A new entry point for C extension modules), which was accepted for Python
3.15.

Since existing ways of defining modules use API that this PEP removes
(namely, :c:type:`PyModuleDef`), extensions will need to migrate to PEP 793's
new “export hook” when switching to Limited API 3.15.


Runtime ABI checks
Expand All @@ -185,6 +190,12 @@ will continue to be responsible for not putting incompatible extensions on
Python's import paths.
This decision makes sense since tools typically have much richer metadata than
what CPython can check.
Typically, build tools and installers use `PyPA packaging metadata`_ and
`platform compatibility tags`_ to communicate compatibility details, but other
models are possible.

.. _PyPA packaging metadata: https://packaging.python.org/en/latest/specifications/core-metadata/
.. _platform compatibility tags: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/

However, CPython will add a line of defense against outdated or misconfigured
tools, or human mistakes, in the form of a new *module slot* containing
Expand Down Expand Up @@ -218,15 +229,45 @@ The ``abi3t`` wheel tag
-----------------------

Wheels that use a stable ABI compatible with free-threading CPython builds
should use a new ABI tag: ``abi3t``.
should use a new `ABI tag`_: ``abi3t``.
The name is chosen to reflect the fact that this ABI is similar to ``abi3``,
except changes necessary to support free-threading (which uses the letter ``t``
in existing, version-specific ABI tags like ``cp314t``).
with limitations necessary to support free-threading (which uses the letter
``t`` in existing, version-specific ABI tags like ``cp314t``).

.. _ABI tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#abi-tag

Package installers should treat this tag as completely separate from ``abi3``.
They should allow ``abi3t``-tagged wheels for free-threaded builds wherever
they currently allow ``abi3``-tagged ones for (orherwise equal) non-free-threaded
builds.

Build tools should generate ``abi3.abi3t`` instead of ``abi3`` when the Python
tag is ``cp315`` and above (or equivalently: when setting ``Py_LIMITED_API``
to ``3.15`` (``0x030f0000``) or above).
``abi3.abi3t`` is a `compressed tag set`_ that signals compatibility with both
``abi3`` and ``abi3t``.

.. _compressed tag set: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#compressed-tag-sets

.. note::

The version of the Stable ABI is indicated by the `Python wheel tag`_; this
PEP does not change that.
For example, a wheel tagged ``cp315-abi3.abi3t`` will be compatible with
3.15, 3.16, and later versions;
``cp317-abi3.abi3t`` will be compatible with 3.17+.

Since wheels built using Limited API 3.15 will be compatible with both
GIL-enabled builds and free-threaded ones, they should use the
`compressed ABI tag set <https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#compressed-tag-sets>`__
``abi3.abi3t``.
.. _Python wheel tag: https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/#python-tag

The ``abi3t`` tag can be used in extensions compatible with earlier versions of
free-threaded Python.
For example, an extension compatible with GIL-enabled *and* free-threaded
builds of *3.14*, 3.15, and higher versions would be tagged
``cpy314-abi3.abi3t``.
This PEP does not propose an official way to build such extensions, but since
a mechanism for that can be added later (see
:ref:`a Rejected idea <pep803-no-shim>`), installers should be ready to accept
the tag.


New API
Expand Down Expand Up @@ -297,6 +338,20 @@ free-threaded one.
* ❌
* ✅
* ❌
* * ``cp314-abi3t`` (*)
* ❌
* ✅
* ❌
* ✅
* ❌
* ✅
* * ``cp314-abi3.abi3t`` (*)
* ✅
* ✅
* ✅
* ✅
* ✅
* ✅
* * ``cp315-cp315``
* ❌
* ❌
Expand All @@ -318,6 +373,13 @@ free-threaded one.
* ❌
* ✅
* ❌
* * ``cp315-abi3t`` (*)
* ❌
* ❌
* ❌
* ✅
* ❌
* ✅
* * ``cp315-abi3.abi3t``
* ❌
* ❌
Expand All @@ -326,6 +388,8 @@ free-threaded one.
* ✅
* ✅

(*): Wheels with these tags cannot be built; see table below

The following table summarizes which wheel tag should be used for an extension
built with a given interpreter and ``Py_LIMITED_API`` macro:

Expand All @@ -349,6 +413,14 @@ built with a given interpreter and ``Py_LIMITED_API`` macro:
* 3.14+ (GIL)
* ``PY_PACK_VERSION(3, 14)``
* existing
* * ``cp314-abi3t``
* N/A
* N/A
* out of spec
* * ``cp314-abi3.abi3t``
* N/A
* N/A
* reserved
* * ``cp315-cp315``
* 3.15 (GIL)
* (unset)
Expand All @@ -361,6 +433,10 @@ built with a given interpreter and ``Py_LIMITED_API`` macro:
* 3.15+ (GIL)
* ``PY_PACK_VERSION(3, 15)``
* discontinued
* * ``cp315-abi3t``
* N/A
* N/A
* out of spec
* * ``cp315-abi3.abi3t``
* 3.15+ (any)
* ``PY_PACK_VERSION(3, 15)``
Expand All @@ -371,8 +447,16 @@ Values in the *Note* column:
* *existing*: The wheel tag is currently in use
* *continued*: The wheel tag continues the existing scheme
* *discontinued*: The wheel tag continues the existing scheme, but it will
be discouraged. Older tools may still generate it.
be discouraged. Older tools may still generate it. Installers will
continue to accept it, but only for GIL-enabled builds, even though the wheel
would be compatible with free-threaded ones.
* *new*: Proposed in this PEP.
* *reserved*: A mechanism to build a matching extension is not proposed in
this PEP, but may be added in the future.
Installers should be brepared to handle the tag.
* *out of spec*: Should not be used as-is: extensions should be tagged
``abi3.abi3t`` rather than only ``abi3``.
The entry is included for installers that decompose compressed tag sets.


Security Implications
Expand All @@ -398,7 +482,8 @@ This PEP combines several pieces, implemented individually:
``_Py_OPAQUE_PYOBJECT`` macro.
Implemented in GitHub pull request `python/cpython#136505
<https://github.com/python/cpython/pull/136505>`__.
- For ``PyModExport``, see :pep:`793`.
- For ``PyModExport``, see :pep:`793` and
`GitHub issue #140550 <https://github.com/python/cpython/issues/140550>`_.
- A version-checking slot was implemented in GitHub pull request
`python/cpython#137212 <https://github.com/python/cpython/pull/137212>`__.
- A check for older ``abi3`` was implemented in GitHub pull request
Expand Down Expand Up @@ -430,9 +515,27 @@ It would also make the free-threading memory layout of ``PyObject`` part
of the stable ABI, preventing future adjustments.


.. _pep803-no-shim:

Shims for compatibility with CPython 3.14
-----------------------------------------

It’s possible to build a ``cp314-abi3.abi3t`` extenstion – one compatible
with 3.14 (both free-threaded build and default).
There are several challenges around this:

* making it convenient and safe for general extensions
* testing it (as CPython’s test suite doesn’t involve other CPython versions
than the one being tested)

So, providing a mechanism to build such extensions is best suited to an
external project (for example, one like `pythoncapi-compat`_).
It's out of scope for CPython’s C API, and this PEP.

.. _pythoncapi-compat: https://github.com/python/pythoncapi-compat

To sketch how such a mechanism could work:

The main issue that prevents compatibility with Python 3.14 is that with
opaque ``PyObject`` and ``PyModuleDef``, it is not feasible to initialize
an extension module.
Expand All @@ -443,12 +546,11 @@ free-threading and GIL-enabled) are “frozen”, so it is possible for an
extension to query the running interpreter, and for 3.14, use
a ``struct`` definition corresponding to the detected build's ``PyModuleDef``.

This is too onerous to support and test in CPython's Limited API at this point,
but it may be allowed in the future.

.. _pep803-no-avoid-abi3t:

Using the Python wheel tag to determine compatibility
-----------------------------------------------------
Using the Python version wheel tag to determine compatibility
-------------------------------------------------------------

A previous version of this PEP avoided adding a new wheel tag (``abi3t``),
and specified that wheels tagged ``abi3`` would be compatible with
Expand Down Expand Up @@ -477,6 +579,16 @@ an unnecessary technical change.
Using ``abi3.abi4`` in wheel tags but only ``.abi3`` in filenames would
look more inconsistent than ``abi3.abi3t`` and ``.abi3``.

If we add ``abi4`` tag, the ``Py_LIMITED_API`` value would either need to:

* change to start with ``4`` to match ``abi4``, but no longer correspond
to ``PY_VERSION_HEX`` (making it harder to generate and check), or
* not change, making it inconsistent with ``abi4``.

Adding ``abi3t`` is a smaller change than adding ``abi4``, making it work
better as a transitional state before larger changes like :pep:`809`'s
``abi2026``.


Copyright
=========
Expand Down