[DO NOT MERGE] smoke test cryptography with CFFI 1.16.0 pre-release

[nitzmahone]

• testing cryptography with a hard dep on CFFI 1.16.0rc1 where possible - we dropped Python 3.7 support but added python_requires metadata accordingly, so 3.7 should still use CFFI==1.15.1.

[nitzmahone]

Welp, looks like there's going to be a CFFI 1.16.0rc2, and probably some new surrogate test coverage once I figure out what we screwed up with the shim module. ;) I'll update this to force rc2 once we get that sorted.

[alex]

Sounds good, thanks for testing.

[nitzmahone]

OK, rc2 is looking a lot better- all CI tests are green (CFFI's source layout masked a packaging failure, even though all tests run against the built wheels). The wheel builder jobs are still failing, but I assume that's maybe due to some isolation/caching going on for the deps? Please let me know if there's more I should look into on the CFFI end, but ATM I'm assuming cryptography is good to go if we release this week.

[alex]

See: .github/requirements/build-requirements.{txt,in} for the wheel builder jobs.

[nitzmahone]

@alex aha, thanks for the tip- that did it. All the non-PyPy wheel builder jobs are green against CFFI 1.16.0rc2, and since it seems to be a "non-user-serviceable" component of PyPy (I've asked at https://foss.heptapod.net/pypy/pypy/-/issues/4005 to be sure there's nothing more we can verify there upstream), I'm feeling pretty good about it.

I'll leave this PR open and validate any further RCs and/or the final bits as they're released.

Also: cryptography is the primary justification for our team to maintain CFFI's packaging on PyPI (since that keeps cryptography easily pip-installable for Ansible users). It looks like the project is steadily migrating to native PyO3 wrappers for the various C libraries linked under hazmat, which IIUC would no longer require CFFI once complete. Two questions:

1. Am I understanding correctly the decreasing need for cffi?
2. Is there a planned timeline for that migration to be complete? (I know, silly question of an open source project)

I know there's an awful lot of CFFI code stubs remaining- if the technical issues for an all-PyO3-no-CFFI cryptography have been solved, and it's merely an issue of people-power to rewrite all the bindings, I'd be interested to discuss that, as I'd much prefer to "write myself out of a job" maintaining CFFI if possible. 😉

[reaperhulk]

We are slowly decreasing our usage of cffi, but pyOpenSSL uses our bindings layer and what we do there is still very much TBD.

Outside of the pyOpenSSL situation I'm not aware of any fundamental technical issues, but as we convert we run into many yaks that are in need of shaving. Frequently that's additions to rust-openssl, but occasionally it ranges farther afield into the PyO3 or the broader Rust ecosystem.

I don't think we can really even guess a date until we get to the point where we have a plan for pyOpenSSL, but @alex may have a different opinion 😄

[alex]

Nope, that's totally in line with my view. Absent the pyOpenSSL issue I'd say there's another year or so of replacement. But we don't even have a basic concept for pyOpenSSL.

…

On Tue, Sep 26, 2023, 12:30 PM Paul Kehrer ***@***.***> wrote: We are slowly decreasing our usage of cffi, but pyOpenSSL uses our bindings layer and what we do there is still very much TBD. Outside of the pyOpenSSL situation I'm not aware of any fundamental technical issues, but as we convert we run into many yaks that are in need of shaving. Frequently that's additions to rust-openssl, but occasionally it ranges farther afield into the PyO3 or the broader Rust ecosystem. I don't think we can really even guess a date until we get to the point where we have a plan for pyOpenSSL, but @alex <https://github.com/alex> may have a different opinion 😄 — Reply to this email directly, view it on GitHub <#9653 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAAGBG2KKCFX7DOYNMTMMLX4L7LXANCNFSM6AAAAAA5G7AYGQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[nitzmahone]

OK, sounds like we're on the hook for another year or two at least- do please let us know if you get to a point where it's purely a matter of cranking out binding adapters or something, because we could probably justify sending some capable hands your way for a bit if it meant eliminating CFFI from the picture.

[alex]

This change here is what caused the test failures btw.

[nitzmahone]

Ahh, I ran the pip-compile under Python 3.11, so setuptools et al probably decided they didn't need an aftermarket TOML parser.

[alex]

I think we're good. Thanks!