How to build a standalone/frozen meson

[ilg-ul]

I'm maintaining a binary distribution of meson for almost 4 years, and things were functional, until recently, when I tried to build a recent glib and I encountered the following error:

Checking if "/proc/self/cmdline" runs: NO (1)

meson.build:2422:26: ERROR: <PythonExternalProgram 'python3' -> ['/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson']> is not a valid python or it is missing distutils

A full log can be found at /Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py/meson-logs/meson-log.txt
WARNING: Running the setup command as `meson [options]` instead of `meson setup [options]` is ambiguous and deprecated.

Program stderr:

meson.build:2422:26: ERROR: <PythonExternalProgram 'python3' -> ['/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson']> is not a valid python or it is missing distutils

It looks like recent glib releases (2.80.0 and several older versions too, up to a point) use some advanced meson features (introspection?), which are not available in a standalone distribution like mine.

As it can be seen, I use a binary called meson, which is linked to the python library, so the system python3 executable is no longer needed. Until now, this worked fine for a cross-platform distribution (macOS, Linux, Windows).

The meson.c code is relatively straightforward, it seems complicated because it still includes some debug code, but it just initialises the Python environment and executes the mesonmain.main() function.

The error comes from the mesonbuild/modules/python.py file, which invokes meson to run some Python code, and obviously fails.

I'm not familiar with the meson code used by glib, and I don't know if the features used by glib are legit or not, but I think that the meson assumption that arg[0] is the path to the Python interpreter is a bit too strong.

I'd appreciate any advice on how to improve my standalone project to accommodate such advanced use cases too.

[eli-schwartz]

I'm not familiar with the meson code used by glib, and I don't know if the features used by glib are legit or not, but I think that the meson assumption that arg[0] is the path to the Python interpreter is a bit too strong.

It doesn't assume that. It assumes that python's sys.executable is the python interpreter, which is true when python is used as a script shebang regardless of what argv[0] is.

This does not work when you use a custom executable entrypoint which isn't a python interpreter, but rather loads the python interpreter via libpython, which you said you are doing.

Meson does have a fallback based on the notion that meson officially supports two forms of distribution:

• installing as a python script together with an external python where sys.executable is python itself
• PyInstaller, which we use to create Windows and macOS installers, which documents at https://pyinstaller.org/en/stable/runtime-information.html that it sets two attributes: sys.frozen = True and sys._MEIPASS = '/path/to/pyinstaller/bundled/runtime'

In this case, meson attempts to use meson runpython, an internal entrypoint that executes its arguments via runpy.

Supporting additional forms of freezing would require a code change to meson (or for your custom freeze program to imitate the implementation details of PyInstaller).

[eli-schwartz]

I wrote a small change to allow your freeze tooling to just set sys.frozen but not bother with sys._MEIPASS:

master...eli-schwartz:meson:misc-fixes (1465c76)

Would this work for you?

[ilg-ul]

Thank you @eli-schwartz for your answer, it explains why my code fails (I did not set the two attributes).

I don't remember details now, but in 2020 I did not find a freeze tool to work properly in my build environment, so I simply added a meson.c and compiled all .py code:

• https://github.com/xpack-dev-tools/meson-build-xpack/blob/xpack/src/meson.c

I'll try to add an executable with the Python interpreter to my distribution and adjust the startup code to pass the path to it.

I'll also revisit PyInstaller, perhaps this time I can find a solution to use it. The major difficulty is that I build the Windows binaries on Linux, so I need to either find a way to start PyInstaller in cross mode, or run it under wine.

[ilg-ul]

I enabled debug messages on my standalone meson binary, and patched universal.py to no longer check _MEIPASS:

if getattr(sys, 'frozen', False):
    # using a PyInstaller bundle, e.g. the MSI installed executable
    python_command = [sys.executable, 'runpython']
else:
    python_command = [sys.executable]
print('universal.py:python_command: ', python_command)
_meson_command: T.Optional['ImmutableListProtocol[str]'] = None

The glib meson setup command is:

[meson setup --prefix /Users/ilg/Work/xpack-dev-tools-build/qemu-riscv-8.2.2-1/darwin-x64/x86_64-apple-darwin23.3.0/install --includedir /Users/ilg/Work/xpack-dev-tools-build/qemu-riscv-8.2.2-1/darwin-x64/x86_64-apple-darwin23.3.0/install/include --libdir /Users/ilg/Work/xpack-dev-tools-build/qemu-riscv-8.2.2-1/darwin-x64/x86_64-apple-darwin23.3.0/install/lib --backend ninja -D tests=false -D selinux=disabled /Users/ilg/Work/xpack-dev-tools-build/qemu-riscv-8.2.2-1/darwin-x64/x86_64-apple-darwin23.3.0/build/glib-2.80.0 /Users/ilg/Work/xpack-dev-tools-build/qemu-riscv-8.2.2-1/darwin-x64/sources/glib-2.80.0]

The setup code does a lot of actions, but at the end it fails with:

Checking if "/proc/self/cmdline" runs: NO (1)
Could not introspect Python (['/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson', 'runpython', '/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py']): exit code 2
Program stdout:

argc: 3
argv[0]: '/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson'
argv[1]: 'runpython'
argv[2]: '/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py'
self_exe_path: /Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/bin/meson
self_folder_path: /Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/bin
home_folder_path: /Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application
initial Py_GetPythonHome(): (null)
Py_SetPythonHome('/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application')
initial Py_GetPath(): (null)
Py_SetPath('/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12:/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/lib-dynload')
Py_GetProgramFullPath(): (null)
Py_GetPrefix(): 
Py_GetExecPrefix(): 
Py_IsInitialized()
sys.executable:  /Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson
sys.path:  ['/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12', '/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/lib-dynload']
sys.frozen:  True
sys.argv:  ['/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/bin/meson', 'runpython', '/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py']
universal.py:python_command:  ['/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson', 'runpython']

Program stderr:

Traceback (most recent call last):
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/mesonmain.py", line 186, in run
    return options.run_func(options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/mesonmain.py", line 141, in run_runpython_command
    runpy.run_path(options.script_file, run_name='__main__')
  File "<frozen runpy>", line 286, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py", line 64, in <module>
    paths, install_paths = get_install_paths()
                           ^^^^^^^^^^^^^^^^^^^
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/mesonbuild/scripts/python_info.py", line 59, in get_install_paths
    paths = sysconfig.get_paths(scheme=scheme)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/sysconfig.py", line 626, in get_paths
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/sysconfig.py", line 270, in _expand_vars
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/sysconfig.py", line 723, in get_config_vars
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/sysconfig.py", line 670, in _init_config_vars
  File "/Users/ilg/Work/xpack-dev-tools-build/meson-build-1.4.0-1/darwin-x64/application/lib/python3.12/sysconfig.py", line 536, in _init_posix
ModuleNotFoundError: No module named '_sysconfigdata__darwin_darwin'


meson.build:2422:26: ERROR: <PythonExternalProgram 'python3' -> ['/Users/ilg/Work/xpack-dev-tools/qemu-riscv-xpack.git/build/darwin-x64/xpacks/.bin/meson', 'runpython']> is not a valid python or it is missing distutils

So meson correctly invokes itself with runpython, as you explained.

If I read the error message correctly, the actual reason why it failed is the missing sysconfig module.

Is this a meson dependency that needs to be installed separately?

My current Python environment is pristine, as installed from the Python sources, and in the resulting python3.12 folder I see a sysconfig.py file, but I have no idea how to satisfy the missing _sysconfigdata__darwin_darwin requirement.

[eli-schwartz]

The distutils warning really happens because it has been assumed for a long time that this code is unlikely to fail except on Debian when the distutils module is stripped out by Debian.

The missing _sysconfigdata__darwin_darwin module implies that python itself has a bug in how it was installed, since that file should have been created by the python build system when installing python, on any POSIX platform including darwin that uses the ./configure && make install to build cpython (i.e. not the Windows Visual Studio solution files).

[eli-schwartz]

For example on Linux, my /usr/lib/python3.12/ installation of python has the file:

/usr/lib/python3.12/_sysconfigdata__linux_x86_64-linux-gnu.py

[ilg-ul]

The missing _sysconfigdata__darwin_darwin module implies that python itself has a bug in how it was installed, since that file should have been created by the python build system when installing python

Oops! You're right, not all files were copied to the distribution. I fixed this.

Now python_info.py returns a large JSON. Unfortunately it does not pass some validation, and the result is still ... is not a valid python or it is missing distutils.

python_info.json

Is there anything special in this file that meson mandates?

[eli-schwartz]

Can you show the full meson-log.txt?

[ilg-ul]

The macOS standalone build of meson 1.4.0 seems functional now, I was able to build glib and qemu, which are the most complex projects using meson build that I maintain. I expect the Linux build to be ok too, so half of the problem is solved. The other half, the Windows binaries, will take some more time.

The reason why the JSON validation failed were some unexpected debug prints to stdout instead of stderr.

There are a few issues that I will enter as separate comments.

[ilg-ul]

Pretend to be a PyInstaller binary vs a meson patch to ignore sys._MEIPASS?

My standalone binaries use a meson executable that is neither a python script nor are created by PyInstaller, so the current meson fails to run Python code.

To overcome this, I implemented two solutions, but I'm not sure which one is preferred.

The first was to patch meson to ignore sys._MEIPASS, as you suggested in 1465c76

        sed -i.bak \
          -e "s|if getattr(sys, 'frozen', False) and hasattr(sys, '_MEIPASS'):|if getattr(sys, 'frozen', False):|" \
          "${meson_src_folder_name}/mesonbuild/utils/universal.py"

This works fine for my builds, but I don't know the implications of having this patch integrated into the upstream meson.

The second solution is to pretend my standalone binaries are generated by PyInstaller, and add the sys._MEIPASS path.

In my case the resulting executable is bin/meson and the Python code is in lib/python3.12. This is slightly different from the folder structure used by PyInstaller.

This also works fine with the current meson which does not seem to care of the content of the _MEIPASS folder, but I don't know the implications for possible projects using this definition to access the Python internals.

Of these two solutions my favourite is the second, since I don't have to keep the patch up to date with future meson versions.

@eli-schwartz, can you suggest your favourite solution?

[eli-schwartz]

I included my suggestion in #13017 which should mean (once it is merged) you don't have to keep the patch up to date with future meson versions.

Let's see what others think about it :)

[ilg-ul]

Preserve the Python source code vs compiling everything in frozen distributions?

My previous solution for the standalone meson package was to compile all .py code into .pyc and distribute only the compiled code.

Apparently PyInstaller takes, at least with the default configuration, a similar approach.

However, as discovered while building glib, there are cases when meson needs to run scripts available in the mesonbuild/scripts folder, like python_info.py.

This fails if all Python files are compiled and the original sources are no longer included in the distribution.

Currently I updated the build scripts to keep both source and compiled Python code, but the resulting binaries are pretty large now.

I would like to remove the source files that are not needed, but knowing which is which does not seem an obvious choice.

@eli-schwartz, in the meson folder structure, are there any other files except those in the scripts folder that need to be present in source form?

Do you know of such scripts in the standard Python library that need to be present in source form?

[eli-schwartz]

meson/packaging/hook-mesonbuild.py

Line 24 in a63739d

datas += collect_data_files('mesonbuild.scripts', include_py_files=True, excludes=['**/__pycache__'])

This is how we collect everything with PyInstaller. There's a couple directories which contain non-python files, which of course need to be accessible, and the scripts directory specifically treats all .py files inside as data files that need to be available as arbitrary data files.

Not all of them are actually needed by the way, but it was easier to not continually maintain a list of the ones needed.

The reason the ones which are needed, are needed, is because they are accessed as files and used in commands that literally run `python path/to/scripts/somescript.py and of course that doesn't work with .pyc files instead.

[eli-schwartz]

If we ever add more files that need this special handling I would want to add them to scripts/ as well so I don't need to keep updating the PyInstaller packaging code.

[ilg-ul]

Your script is a bit too complicated for my poor understanding of PyInstaller. Do you confirm that I can compile everything except the meson scripts folder?

[eli-schwartz]

You can compile everything, even the scripts folder. Just as long as you don't delete the .py files from the scripts folder.

[ilg-ul]

Additional Python dependencies that might be useful during meson builds?

Another small problem I had to fix was a dependency to the packaging Python module.

I did not investigate if this is a dependency needed by the meson code itself or by the glib meson code, I simply added a pip install packaging and the glib build passed.

For just in case I also added a pip install setuptools, but I confess that I have no idea what this module does, I only read some comments that some internal Python functionality was moved outside the Python core first to distutils and now to setuptools.

@eli-schwartz, given that my distribution is standalone/frozen, and it is not possible to install additional packages, is there a list of recommended Python packages to have at hand during meson builds?

[eli-schwartz]

Meson doesn't use any external packages, but people writing scripts that run as custom commands during the build may be using meson's bundled python.

If there's an external python3 command available on $PATH they will use that -- if not, meson is helpful and injects itself.

So the question is what third-party modules might be used by authors of scripts.

The official MSI installer for meson on Windows doesn't include any third-party modules so that probably encourages users to avoid overreliance on such modules. However the packaging module does contain the version parsing and comparison code that replaces distutils functions, so that's probably a good thing to include just in case as people may have ported away from distutils in their script commands.

[ilg-ul]

Meson doesn't use any external packages, but people writing scripts that run as custom commands during the build may be using meson's bundled python.

Yes, that's the reason I prefer to bundle the full Python library, not only the modules used by meson, as PyInstaller does.

the packaging module ... probably a good thing to include

Great, thank you for confirming.

I guess setuptools is not really necessary and can be ignored, right?

[ilg-ul]

I managed to rework my build scripts and the resulting standalone meson passes complex builds, like recent glib and qemu.

I could not use PyInstaller, mainly due to some macOS specific issues.

In case someone will be interested in details, the xPack standalone distribution uses a meson/meson.exe executable, compiled from meson.c and linked to the shared library libpython; on Linux/macOS this library is compiled from sources; on Windows the library is available in a separate embedded package (like python-3.X.Y-mbed-amd64.zip).

On Linux/macOS, the Python library must include the _sysconfigdata_* module, which is dynamically generated for each platform and not present directly in the source package. On Windows, the Python library is created from the modules available in the embedded Python archive.

Except the test folder which provides modules used to test Python itself and can be removed, the entire library is preserved in the final distribution, to allow advanced meson applications to execute any legal Python code.

Even more, in addition to pip and mesonbuild, two more modules (packaging and setuptools) are also installed via pip and preserved in the final library.

To save space, all .py files are replaced by their .pyc compiled equivalents.

The original .py files in the mesonbuild/scripts are preserved in the distribution, being invoked as scripts by meson.

The PyInstaller specific definitions (sys.frozen and sys._MEIPASS) are defined, although the content of the _MEIPASS folder is slightly different compared to that used by PyInstaller. A third definition (sys.is_xpack) was added to mark the xPack distribution.