Enable build of linux-aarch64 wheels

[jonded94]

Currently, only Mac Intel/ARM, Windows x86_64 and Linux x86_64 wheels are being built in this repository.

It would be nice if Linux ARM wheels could also be available, for example:

• native Docker containers on Mac
• modern ARM-based EC2 instances on AWS
• other ARM-native machines

For that I prepared this PR.

Disadvantages:

libc requirements

Because of

• Cannot use grpcio on manylinux2014_aarch64 (undefined symbol) grpc/grpc#33734
• grpc_tools.protoc undefined symbol: _ZTVNSt7__cxx1119basic_istringstreamIcSt11char_traitsIcESaIcEEE grpc/grpc#30218

I was unable to get this to work with manylinux2014 and had to switch all the way to 2_28 since 2_24 is deprecated and EOL.

This obviously has the implication that the wheel no longer consumable by OS's that have a libc < 2_28. This should not be of concern for ARM machines at least since one could expect that they run on relatively new hardware/OS's anyways. Don't know how you feel about that though! One could of course build Linux x86 wheels on the older manylinux version, but that would slightly increase complexity in the Dockerfile.

CI times

I noticed a rough ~5x increase of CI times since all the ARM Wheel builds and tests are run through QEMU which is notoriously slow.

[jonded94]

Note that I had to make changes to the Dockerfile, I pointed the Linux build containers that you use for now to ghcr.io/jonded94/resiliparse-manylinux_2_28_aarch64. If you are interested in merging this, this should be changed of course before that.
The Dockerfiles we're built manually, proper jobs for building these are missing here.

EDIT: Solved with merged PR #37

[wumpus]

This is a great idea. It so happens that the Common Crawl Foundation mostly uses linux ARM cloud machines because they are cheaper to rent. Thank you for this PR.

[jonded94]

Hey :) What is the status on this PR?

Not having linux ARM wheels still blocks usage of FastWARC inside ARM-based AWS instances and native Docker containers on M1 Macs.

Is there anything I can do? Maybe skip testing of the ARM wheels, only build them through QEMU?

[phoerious]

Hi sorry,
I've been too busy to look into it further. I would definitely skip the tests and ideally cross-compile natively. But we can also virtualise it for the time being.

[jonded94]

Since I stumbled upon a weird issue (pypa/cibuildwheel#1771) where cibuildwheel suddenly refuses to execute the new aarch64 manylinux container in the correct platform mode (but tries to do it in AMD64 and fails to find a fitting image), I had to separate this build job for now. But that seems to work for now. Note that I disabled testing for aarch64 as agreed on.

Note that build-asan is currently also blocked since I forgot to upgrade the libasan gcc toolset to version 12. I created a PR for that: #38

[phoerious]

Great. Is this ready to merge from your part? Could you do a rebase before?

[jonded94]

All is done now; PR should be ready to merge :) Sorry for the long time that this took.

[phoerious]

The first aarch64 builds are up. Please test.
I'm not happy with the build times (took more than an hour), so I've disabled builds for 3.8 and 3.9 for now and trigger the whole aarch64 job only on a tags/ ref.

[jonded94]

Great to hear!

Some idea that I use currently at my company to circumvent the entire cross-compilation situation: One could use the Mac M-series runners and spawn the Manylinux ARM Linux docker container natively in them. I'm not too familiar with which settings one would have to overwrite in cibuildwheel to make that work (forcing Linux builds through OCI containers under MacOS); I usually just build the compilation-, auditwheel- and test-step manually anyways without using cibuildwheel.

That should reduce the build times back to <=5min I guess.

[phoerious]

That would probably be as easy as just running everything inside the Linux Docker image on the macOS-14 runner. A disadvantage would be that you cannot easily express that as a simple build matrix.

My last experiments with the native M1 runners failed (for other reasons), but it might be worth trying that again.