Support the free-threaded build of CPython

[ngoldbaum]

Feature request

Right now safetensors supports the free-threaded build in principle because it uses PyO3 0.23, but doesn't explicitly declare support. This means if you install it on the free-threaded build, Python prints a warning that it is re-enabling the GIL at runtime:

goldbaum at Mac in ~/Documents/safetensors on main
± pip install safetensors
Collecting safetensors
  Downloading safetensors-0.5.2.tar.gz (66 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: safetensors
  Building wheel for safetensors (pyproject.toml) ... done
  Created wheel for safetensors: filename=safetensors-0.5.2-cp313-cp313t-macosx_11_0_arm64.whl size=413610 sha256=ba806bc9fd2250873da3cafc58fc502321abc30df2d77c66b096aba4f709cc4d
  Stored in directory: /Users/goldbaum/Library/Caches/pip/wheels/4a/50/b2/61d052951768fd300fda20bfa3bc357184d4be288878086c21
Successfully built safetensors
Installing collected packages: safetensors
Successfully installed safetensors-0.5.2

goldbaum at Mac in ~/Documents
○  python
Python 3.13.1 experimental free-threading build (main, Dec 10 2024, 14:07:41) [Clang 16.0.0 (clang-1600.0.26.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import safetensors
<frozen importlib._bootstrap>:488: RuntimeWarning: The global interpreter lock (GIL) has been enabled to load module 'safetensors._safetensors_rust', which has not declared that it can run safely without the GIL. To override this behavior and keep the GIL disabled (at your own risk), run with PYTHON_GIL=0 or -Xgil=0.

Motivation

It should be possible and safe to use safetensors on the free-threaded build in effectively single-threaded contexts as people already do on the GIL-enabled build. I haven't dived into the safetensors internals to see what happens when state is shared between threads, but it should also hopefully be possible to detect situations like that and at a minimum generate a runtime error.

Additionally, users on the free-threaded build should be able to get pre-compiled wheels without needing a compiler toolchain to install it.

One significant wrinkle is that the free-threaded build doesn't yet support building extensions using the limited API, so you'll need to build version-specific free-threaded wheels if you want to upload wheels to PyPI.

Your contribution

I am a PyO3 maintainer and have helped other Rust libraries that depend on PyO3 to ship free-threaded wheels. I'm happy to help out here but am a newcomer to the codebase.