Sync mypy including the stubs #14

ilevkivskyi · 2025-11-17T16:57:10Z

No description provided.

Vendor optimized base64 implementation from https://github.com/aklomp/base64. This is based on commit 9e8ed65048ff0f703fad3deb03bf66ac7f78a4d7 (May 2025). Enable SIMD on macOS (64-bit ARM only). Other platforms probably use a generic version. I'll look into enabling SIMD more generally in a follow-up PR. A `b64encode` micro-benchmark was up to 11 times faster compared to the stdlib `base64` module (on a MacBook Pro).

mr-c · 2025-11-17T21:52:19Z

lib-rt/setup.py

    if compiler.compiler_type == "unix":
        cflags += ["-O3"]
+        if X86_64:
+            cflags.append("-msse4.2")  # Enable SIMD (see also mypyc/build.py)


No auto detection? I'll have to patch this out for Debian, the amd64 baseline is sse2

Suggested change

cflags.append("-msse4.2") # Enable SIMD (see also mypyc/build.py)

There is autodetection already, -msse4.2 is not needed: https://github.com/mypyc/librt/pull/14/files#diff-25f140cc37c41f9eb31ca08bca258a6bb6f86e404efde9344762e6aaac4d5494R188-R203

Looking at the original mypy PR python/mypy#20244 that added this, I guess it was necessary for some compilers, maybe @JukkaL can clarify.

The build fails without this, since SSE4.2 features can't be used without the flag, at least on Ubuntu 24.04. In particular, the C files within arch/sse in the path may need this flag. Auto-detection is supported by the underlying base64 library, but we'd still need to compile the SSE-related files using this flag, and setuptools doesn't make it straightforward to have different C compiler flags for specific files.

Here is a fix for per-file compile flags python/mypy#20253

@ilevkivskyi @JukkaL The fix for per-file base64 compilation flags is ready for review & merging python/mypy#20253

JukkaL · 2025-11-28T14:47:17Z

The per-file compilation flag PR was merged and this PR can be updated to reflect it (Ihttps://github.com/python/mypy/pull/20253).

ilevkivskyi · 2025-11-28T15:02:14Z

OK, I will probably re-run the sync to also include other recent changes to lib-rt.

mr-c · 2025-11-28T17:14:36Z

For the "pyodide" error

emcc: error: passing any of -msse, -msse2, -msse3, -mssse3, -msse4.1, -msse4.2, -msse4, -mavx, -mavx2, -mfpu=neon flags also requires passing -msimd128 (or -mrelaxed-simd)!

I recommend using the neon64 codec + msimd128, as those are automatically mapped well to wasm's simd using the SIMD Everywhere code copy inside of emscripten.

mr-c · 2025-11-28T17:15:17Z

I also recommend making wasm64 wheels as well as the existing wasm32 wheels

ilevkivskyi · 2025-11-28T17:36:07Z

wasm64 is not supported by cibuildwheel, at least it is not the "official table" in https://cibuildwheel.pypa.io/en/stable/options/#build-skip. Also WASM wheels are really experimental as PyPI doesn't support them at all, for now we are just trying to keep things ready for when they will be better supported by PyPA infrastructure.

mr-c · 2025-11-28T17:51:06Z

Here's a patch to fix the pyodide build (and use the NEON SIMD intrinsics on this route)

0001-fix-build-for-pyodide.patch

mr-c · 2025-11-28T17:52:27Z

wasm64 is not supported by cibuildwheel, at least it is not the "official table" in https://cibuildwheel.pypa.io/en/stable/options/#build-skip. Also WASM wheels are really experimental as PyPI doesn't support them at all, for now we are just trying to keep things ready for when they will be better supported by PyPA infrastructure.

I see now that pyodide doesn't support wasm64 yet: https://pyodide.org/en/stable/usage/api/js-api.html#pyodide.LockfileInfo.arch

ilevkivskyi · 2025-11-28T18:06:20Z

Here's a patch to fix the pyodide build (and use the NEON SIMD intrinsics on this route)

Could you make a PR to mypy first (if you have time)? Normally things should go there first, I (or Jukka will then sync those here).
Also it looks like now there is another error:

clang: error: unsupported option '-mavx2' for target 'wasm32-unknown-emscripten'

Not sure if your fix is affected by this.

mr-c · 2025-11-28T18:12:54Z

Here's a patch to fix the pyodide build (and use the NEON SIMD intrinsics on this route)

Could you make a PR to mypy first (if you have time)? Normally things should go there first, I (or Jukka will then sync those here).

Sure, though there is no pyodide CI in the mypy repo

python/mypy#20316

Also it looks like now there is another error:
clang: error: unsupported option '-mavx2' for target 'wasm32-unknown-emscripten'
Not sure if your fix is affected by this.

My fix doesn't have that problem, no. I tested locally using pyodide build

ilevkivskyi · 2025-11-28T18:17:25Z

Sure, though there is no pyodide CI in the mypy repo

We used to build WASM wheels for mypy, they are temporarily disabled (because mypy now depends on librt, so things get tricky without PyPI support for WASM), but as I said it is better keep things ready :-)

ilevkivskyi · 2025-11-30T00:11:01Z

Hm, @mr-c now there are some weird compile errors in WASM build:

  In file included from base64/arch/neon64/codec.c:59:
  base64/arch/neon64/enc_reshuffle.c:22:15: error: call to undeclared function 'vsliq_n_u8'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
     22 |         out.val[1] = vsliq_n_u8(out.val[1], in.val[0], 4);
        |                      ^
  base64/arch/neon64/enc_reshuffle.c:22:13: error: assigning to 'simde_uint8x16_t' (vector of 16 'uint8_t' values) from incompatible type 'int'
     22 |         out.val[1] = vsliq_n_u8(out.val[1], in.val[0], 4);
        |                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  base64/arch/neon64/enc_reshuffle.c:23:13: error: assigning to 'simde_uint8x16_t' (vector of 16 'uint8_t' values) from incompatible type 'int'
     23 |         out.val[2] = vsliq_n_u8(out.val[2], in.val[1], 2);
        |                    ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  3 errors generated.

ilevkivskyi · 2025-11-30T00:19:11Z

Interestingly, the errors only appear in Python 3.12 WASM build, Python 3.13 works fine.

mr-c · 2025-11-30T00:20:51Z

Interestingly, the errors only appear in Python 3.12 WASM build, Python 3.13 works fine.

Ah, I was testing with Python 3.13. I will look into this soon

mr-c · 2025-12-01T14:13:46Z

@ilevkivskyi Fix for Pyodide with Python 3.12 is at python/mypy#20342

Fixes the issue reported at mypyc/librt#14 (comment) For Python 3.12, [the pyodide_2024_0 ABI requires emscripten 3.1.58](https://pyodide.org/en/stable/development/abi.html#pyodide-2024-0), which uses an older version of SIMDe that doesn't include all the NEON 64-bit intrinsics we need. So we use the 32-bit NEON intrinsics implementation of base64 for Pyodide on Python 3.12, and use the 64-bit NEON intrinsics implementations of base64 for Pyodide on Python 3.13. [Pyodide for Python 3.13, the pyodide_2025_0 ABI, uses emscripten 4.0.9](https://pyodide.org/en/stable/development/abi.html#pyodide-2025-0-under-development) Emscripten versdion 3.1.63 appears to be the first version with the updated SIMDe versions: emscripten-core/emscripten@6223c43

ilevkivskyi added 4 commits November 17, 2025 16:54

Sync mypy including the stubs

7ab7ea7

Update build script

a54fc19

Update build script

a771e77

Try fixig build

2fed789

ilevkivskyi mentioned this pull request Nov 17, 2025

Ship stub files with librt python/mypy#20245

Closed

ilevkivskyi added 2 commits November 17, 2025 19:38

Try fixing WASM builds

111e8a1

Try another way

6b8d939

ilevkivskyi mentioned this pull request Nov 17, 2025

1.19 Release Planning python/mypy#19964

Open

mr-c reviewed Nov 17, 2025

View reviewed changes

ilevkivskyi added 3 commits November 18, 2025 11:37

Update the license as well

d3c1178

Merge remote-tracking branch 'origin/master' into add-stubs

613a106

Bump version

3f0c469

Re-run sync script with latest mypy master

2bcc37e

Re-apply old wasm fix

7c4a4cc

ilevkivskyi added 2 commits November 29, 2025 22:58

Merge branch 'master' into add-stubs

d64016e

Run another sync to get Pyodide fix

08670ac

mr-c mentioned this pull request Dec 1, 2025

[mypyc] lib-rt base64: support pyodide for Python 3.12 python/mypy#20342

Merged

mr-c mentioned this pull request Dec 1, 2025

test build for https://github.com/python/mypy/pull/20342 ; do not merge #19

Closed

One more sync for Pyodide

acf4fe1

mr-c approved these changes Dec 1, 2025

View reviewed changes

Sync mypy including the stubs #14

Are you sure you want to change the base?

Sync mypy including the stubs #14

Uh oh!

Conversation

ilevkivskyi commented Nov 17, 2025

Uh oh!

mr-c Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

mr-c Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

ilevkivskyi Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

JukkaL Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

mr-c Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

mr-c Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

JukkaL commented Nov 28, 2025

Uh oh!

ilevkivskyi commented Nov 28, 2025

Uh oh!

mr-c commented Nov 28, 2025

Uh oh!

mr-c commented Nov 28, 2025

Uh oh!

ilevkivskyi commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mr-c commented Nov 28, 2025

Uh oh!

mr-c commented Nov 28, 2025

Uh oh!

ilevkivskyi commented Nov 28, 2025

Uh oh!

mr-c commented Nov 28, 2025

Uh oh!

ilevkivskyi commented Nov 28, 2025

Uh oh!

ilevkivskyi commented Nov 30, 2025

Uh oh!

ilevkivskyi commented Nov 30, 2025

Uh oh!

mr-c commented Nov 30, 2025

Uh oh!

mr-c commented Dec 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ilevkivskyi commented Nov 28, 2025 •

edited

Loading