support for building other's wheel or any wheel

[pombredanne]

I maintain several projects that have native code that I want to build with cibuildwheel and there are also several projects that I do not maintain for which I also want to build wheels that be missing entirely or only for some os/arch/python combos. These end up either:

• in PyPI for the projects I maintain
• in an secondary PyPI simple index for use as a pip --extra-index-url for the projects that I do not maintain (pushed first in https://github.com/nexB/thirdparty-packages/tree/main/pypi and published to my own PyPI-like index)

Typically the input could be something like a list of pinned name/version requirements (either passed programmatically, as args or a pip requirements.txt file). I use https://github.com/nexB/skeleton/blob/main/etc/scripts/build_wheels.py today that uses Azure to spin multiple jobs and send back Linux/macOS/Windows built wheels

My question would be how to best set things with cibuildwheel to have something similar?
My ideal UX is a CLI that I can pass a --name/version from PyPI, an sdist archive or a git repo and that would spit back built wheels for all the os/arch/python (passed as args too)
Side question: Is this something that you would see integrated in cibuildwheel?

I reckon that https://github.com/astrofrog/autowheel may be a base but this looks like a different use case though could be used as a base.

[pombredanne]

One approach could be to point to a directory that only contains a requirements.txt file?

Of course I could create a dummy setup with install_requires ... but this will not work: I am interested in build the install requires and cibw will complain and fail if the requiring dummy wheel is not platform-specific with "Build failed because a pure Python wheel was generated."

So I guess part of my question is also how to build all the dependent wheels of a given package (be they "pure" or not)

[pombredanne]

Just to clarify my use case: given projects that use many native deps, I want an easy way to get pre-built wheels for all of these deps, on Linux, macOS and Windows (or pure), whether or not these exist on PyPI or not and this with maximum automation and optimal compatibility. ... in particular to avoid these kind of head scratchers pypa/pip#10824

[pombredanne]

@joerick @YannickJadoul would you be interested to get a "build any wheel" feature contributed to this tool? Or should I instead work out something completely separate?

[joerick]

My feeling is that the best way to get this functionality would be #597 - add the ability for cibuildwheel to build from a sdist, rather than a source directory - and then create an additional tool that wraps cibuildwheel and does the PyPI interaction bit. Does that make sense? so something like:

┌────────────┐
│            │
│    PyPI    │
│            │
└────────────┘
     ▲  │
     │  │downloads
     │  │
   ┌────┴───────────────────────┐
   │                            │
   │   cibuildwheel-conveyor    │
   │                            │
   └────┬──────┬────────────────┘
        │      ▼ calls
        │     ┌──────────────┐
        │     │              │
        │     │ cibuildwheel │
        │     │              │
        │     └───────────┬──┘
        │      ▲          │
        ▼      │          ▼
       ┌───────┴┐        ┌────────┐
       │ sdist  │        │ wheels │
       └────────┘        └────────┘

The reason for the separate tool is just to keep cibuildwheel focused to the building, as interactions from cibuildwheel to PyPI itself starts to feel like feature bloat to me.

[henryiii]

#597 Would probably be the way to go.

[pombredanne]

@joerick @henryiii Thanks for the replies!

This makes sense... so to recap we could have:

1. cibuildwheel updated to work from sdist Idea: build from SDists #597
2. a new tool frontend that could :
   2.1 build/fetch sdists from various sources including dir or fetch from PyPI and feed cibuildwheel
   2.2 build/fetch pure wheels

I could feed this new tool with a requirements file (FWIW I have a decent parser for these) https://github.com/nexB/pip-requirements-parser and it would return built native wheels, pure wheels and sdists. This would neatly handle the local directory cases

[henryiii]

Isn’t this pip download?

[pombredanne]

Isn’t this pip download?

@henryiii It should and may be could.... but I have experienced mixed results with pip download when the goal is to download pre-built wheels for python versions/platforms/abis/implementations that are not the ones of the running interpreter but rather the ones of the target so I ended up implementing this separately... I should revisit this alright!

So this may become:

• cibuildwheel updated to work from sdist Idea: build from SDists #597
• a new tool frontend that could :
  • build/fetch pure wheels using pip
  • build/fetch sdists from various sources including dir or fetch from PyPI and feed cibuildwheel. The fetch would wrap pip.

There is still an issue: since cibuildwheels doe s not want to build pure wheels, such a frontend needs to implement the logic to determine if a wheel is going to be pure or not BEFORE calling cibuildwheels. This means interpreting tags and applying some smarts as a package could have both a pure wheel and wheels with native.

[pombredanne]

For now I am doing these:

• support building from an sdist archive Idea: build from SDists #597 (comment) with the PR Support building from an sdist archive #1020
• support building pure wheels Support building pure Python wheels #1021 with the PR Optionally build a pure wheel #1022

With these two I could write another tool that would have this UX:

• accept as an input pip requirements options (name, name/spec, requirements file, etc.)
• build or fetch sdists for each
• for each sdist, call cibuildwheel and passing it the sdist. Call it with the --pure-wheel option so it does build a wheel in all cases

The only missing part there is how provide the sdist tarball to the cibuildwheel run.
There are essentially two cases:

• this is on PyPI: this could then be a script called before calling cibuildwheel that would pip download the sdist, and return the tar ball filename to pass as a package_dir cibuildwheel argument (using Support building from an sdist archive #1020 )
• this is NOT on PyPI: this is more complex because CI's prefer to work from checkouts. the details are TBD.