Switch to packaging for parsing metadata and support metadata 2.4 (#1180)

* Remove "content" from set of specially handled metadata fields

The "content" field is always added to the form data after the package
metadata has been flattened, thus it is not needed to handle it in the
flattening method. Remove the associated test.

This will allow to tighten typing in a successive commit.

* Remove "attestations" from the set of specially handled metadata fields

The "attestations" field is a string: strings do not need flattening.

* Refactor code a tiny bit

Avoid looking a key up into a set of one element and remove an
indirection through a module global variable. This will make it a bit
easier to extend the flattening logic in successive commits.

* Switch from pkginfo to packaging for parsing distribution metadata

The packaging package is maintained by the PyPA and it is the de-facto
reference implementation for the packaging standards. Using packaging
for parsing metadata guarantees support for the latest metadata
versions.

warehouse, the Python package index implementation used by PyPI, also
uses packaging for parsing metadata. This guarantees that metadata
parsing is the same on the client and server side, for the most
prominent index.

* Enable some more mypy checks

* Move monkeypatching of metadata 2.0 support to a more proper place

It was done in the support code for the wheel file format but it
affects metadata loading from all supported distribution types. Move
it to generic code.

* Accommodate for invalid metadata produced by setuptools

See pypa/setuptools#4759.

Author: dnicolodi

changelog/1180.misc.txt

@@ -0,0 +1,10 @@
+- ``packaging`` is used instead of ``pkginfo`` for parsing and validating
+  metadata. This aligns metadata validation to the one performed by PyPI.
+  ``packaging`` version 24.0 or later is required. Support for metadata
+  version 2.4 requires ``packaging`` 24.2 or later. ``pkginfo`` is not a
+  dependency anymore.
+- With ``packaging`` version 24.2 or later, metadata fields added with
+  metadata version 2.4 as defined by PEP 639 are now sent to the package index
+  when a distribution is uploaded. This results in licensing information to
+  appear correctly on the package page on PyPI when uploading packages using
+  metadata version 2.4.

docs/conf.py

@@ -283,17 +283,12 @@
 intersphinx_mapping = {
     "python": ("https://docs.python.org/3", None),
     "requests": ("https://requests.readthedocs.io/en/latest/", None),
+    "packaging": ("https://packaging.pypa.io/en/latest/", None),
 }
 
 # Be strict about the invalid references:
 nitpicky = True
 
-# TODO: Try to add these to intersphinx_mapping
-nitpick_ignore_regex = [
-    (r"py:.*", r"pkginfo.*"),
-    ("py:class", r"warnings\.WarningMessage"),
-]
-
 # -- Options for apidoc output ------------------------------------------------
 
 autodoc_default_options = {

mypy.ini

@@ -5,8 +5,7 @@ show_traceback = True
 warn_redundant_casts = True
 warn_unused_configs = True
 warn_unused_ignores = True
-; Enabling this will fail on subclasses of untyped imports, e.g. pkginfo
-; disallow_subclassing_any = True
+disallow_subclassing_any = True
 disallow_any_generics = True
 disallow_untyped_calls = True
 disallow_untyped_defs = True

pyproject.toml

@@ -31,7 +31,6 @@ classifiers = [
 ]
 requires-python = ">=3.8"
 dependencies = [
-	"pkginfo >= 1.8.1",
 	"readme-renderer >= 35.0",
 	"requests >= 2.20",
 	"requests-toolbelt >= 0.8.0, != 0.9.0",
@@ -41,7 +40,7 @@ dependencies = [
 	"keyring >= 15.1; platform_machine != 'ppc64le' and platform_machine != 's390x'",
 	"rfc3986 >= 1.4.0",
 	"rich >= 12.0.0",
-	"packaging",
+	"packaging >= 24.0",
 	"id",
 ]
 dynamic = ["version"]

tests/fixtures/everything.metadata

@@ -0,0 +1,44 @@
+Metadata-Version: 2.4
+Name: BeagleVote
+Version: 1.0a2
+Platform: ObscureUnix
+Platform: RareDOS
+Supported-Platform: RedHat 7.2
+Supported-Platform: i386-win32-2791
+Summary: A module for collecting votes from beagles.
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Keywords: dog,puppy,voting,election
+Home-page: http://www.example.com/~cschultz/bvote/
+Download-URL: …/BeagleVote-0.45.tgz
+Author: C. Schultz, Universal Features Syndicate,
+        Los Angeles, CA <cschultz@peanuts.example.com>
+Author-email: "C. Schultz" <cschultz@example.com>
+Maintainer: C. Schultz, Universal Features Syndicate,
+        Los Angeles, CA <cschultz@peanuts.example.com>
+Maintainer-email: "C. Schultz" <cschultz@example.com>
+License: This software may only be obtained by sending the
+        author a postcard, and then the user promises not
+        to redistribute it.
+License-Expression: Apache-2.0 OR BSD-2-Clause
+License-File: LICENSE.APACHE
+License-File: LICENSE.BSD
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console (Text Based)
+Provides-Extra: pdf
+Requires-Dist: reportlab; extra == 'pdf'
+Requires-Dist: pkginfo
+Requires-Dist: PasteDeploy
+Requires-Dist: zope.interface (>3.5.0)
+Requires-Dist: pywin32 >1.0; sys_platform == 'win32'
+Requires-Python: >=3
+Requires-External: C
+Requires-External: libpng (>=1.5)
+Requires-External: make; sys_platform != "win32"
+Project-URL: Bug Tracker, http://bitbucket.org/tarek/distribute/issues/
+Project-URL: Documentation, https://example.com/BeagleVote
+Provides-Dist: OtherProject
+Provides-Dist: AnotherProject (3.4)
+Provides-Dist: virtual_package; python_version >= "3.4"
+Dynamic: Obsoletes-Dist
+
+This description intentionally left blank.