From 7aea8d80780d53f46de16db826e976d348d9761c Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Sun, 11 Sep 2022 21:53:27 +1000 Subject: [PATCH 01/16] Initial proposal --- designs/2022-09-12-native-launchers-python.md | 79 +++++++++++++++++++ 1 file changed, 79 insertions(+) create mode 100644 designs/2022-09-12-native-launchers-python.md diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md new file mode 100644 index 0000000..ba25124 --- /dev/null +++ b/designs/2022-09-12-native-launchers-python.md @@ -0,0 +1,79 @@ +--- +created: 2022-09-12 +last updated: 2022-09-12 +status: To be reviewed +reviewers: + - TODO +title: Cross-platform native launchers for Python +authors: + - groodt +--- + + +# Abstract + +This document describes an approach for launching `py_binary` artifacts hermetically using the resolved python toolchain. + + +# Background + +Currently, `py_binary` is non-hermetic and launches inconsistently between platforms. + +On macos and Linux, there is a [python_stub](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt) +that is non-hermetic and requires a "bootstrap" python interpreter on the host. The "shebang" can be overridden, but +a "shebang" is always dependent on the runtime host. + +On Windows, there is a [native launcher](https://github.com/meteorcloudy/bazel/blob/master/src/tools/launcher/python_launcher.cc) +that launches `python.exe` on the host which then launches the `py_binary` with the same `python_stub` as macos and linux. + +Related issues: +* [py_binary with hermetic toolchain requires a system interpreter](https://github.com/bazelbuild/rules_python/issues/691) +* [Neither python_top nor python toolchain works with starlark actions on windows](https://github.com/bazelbuild/bazel/issues/7947#issuecomment-495265016) + +This situation is undesirable because it assumes that the target platform has a bootstrapping python interpreter +available and makes the hermetic python interpreters available with `rules_python` less useful. It is also surprising to +users who expect bazel to output self-contained binary artifacts for a target platform. + +The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the python +interpreter in the runfiles and use that to launch the program. Currently, bazel assumes the target platform will +be able to provide the "bootstrapping" functionality. + + +# Proposal + +Extend the native launcher functionality to all platforms and use it to locate the relevant python interpreter and +python program in the `runfiles` tree to launch the `py_binary`. No assumptions should be made about the target platform. + +In pseudo-code, the proposal is as follows: + +``` +exec(env, runfiles-interpreter, ["interpreter_arg1",], "main.py", ["arg1",]) +``` + +| Token | Description | +| ---------------------- | ----------- | +| env | Dictionary of key-value pairs for the environment of the process | +| runfiles-interpreter | The resolved python toolchain in runfiles | +| ["interpreter_arg1",] | An array of arguments to provide to the python interpreter | +| "main.py" | The python program to launch in runfiles | +| ["arg1",] | An array of arguments to provide to the python program as sys.argv[1:] | + +This native launcher idea has been proposed a few times by bazel contributors and the community: +* [Greg Roodt (Community)](https://github.com/bazelbuild/rules_python/issues/691#issuecomment-1174935972) +* [Yun Peng (Google)](https://github.com/bazelbuild/bazel/issues/7947#issuecomment-495265016) +* [Richard Levasseur (Google)](https://github.com/bazelbuild/rules_python/issues/691#issuecomment-1186379617) + +Some related work has been done that fixes Linux to Windows cross-builds of the Windows launcher. See: [Fix Linux to Windows cross compilation of py_binary, java_binary and sh_binary using MinGW](https://github.com/bazelbuild/bazel/pull/16019) +This proposal would aim to go further and have these launchers available on all platforms to that a Windows to Linux +launcher build could occur (assuming appropriate toolchains). + +Once this proposal is implemented, it would enable cross-builds of hermetic `py_binary` for all major platforms. It +would also remove the complexity introduced by having so many chains of nested execution to launch a python program. + +Finally, while this proposal is specific to python, this solution could perhaps be reused for `java_binary`, `sh_binary` +and perhaps be made available for any custom rules that require an interpreter to launch. + + +# Backward-compatibility + +This proposal could require users to setup a cc toolchain for remote execution. From a5fe5f709b2001b9465d5901f388363a8ffe3224 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Mon, 12 Sep 2022 08:24:05 +1000 Subject: [PATCH 02/16] Update 2022-09-12-native-launchers-python.md --- designs/2022-09-12-native-launchers-python.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index ba25124..36ff258 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -64,8 +64,7 @@ This native launcher idea has been proposed a few times by bazel contributors an * [Richard Levasseur (Google)](https://github.com/bazelbuild/rules_python/issues/691#issuecomment-1186379617) Some related work has been done that fixes Linux to Windows cross-builds of the Windows launcher. See: [Fix Linux to Windows cross compilation of py_binary, java_binary and sh_binary using MinGW](https://github.com/bazelbuild/bazel/pull/16019) -This proposal would aim to go further and have these launchers available on all platforms to that a Windows to Linux -launcher build could occur (assuming appropriate toolchains). +This proposal would aim to go further and have these launchers available on all platforms, including cross_builds where appropriate toolchains are in place. Once this proposal is implemented, it would enable cross-builds of hermetic `py_binary` for all major platforms. It would also remove the complexity introduced by having so many chains of nested execution to launch a python program. From ef27d6556e7ae1b2a2e57acd6e7294e8643c76b4 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Mon, 19 Sep 2022 08:13:29 +1000 Subject: [PATCH 03/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 36ff258..44fdf70 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -35,7 +35,7 @@ available and makes the hermetic python interpreters available with `rules_pytho users who expect bazel to output self-contained binary artifacts for a target platform. The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the python -interpreter in the runfiles and use that to launch the program. Currently, bazel assumes the target platform will +interpreter in the runfiles and use that to launch the program. Currently, Bazel assumes the target platform will be able to provide the "bootstrapping" functionality. From b8702294ead91c4b705b14ffded5644bd12a6975 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:41:15 +1000 Subject: [PATCH 04/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 44fdf70..121efcf 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -31,7 +31,7 @@ Related issues: * [Neither python_top nor python toolchain works with starlark actions on windows](https://github.com/bazelbuild/bazel/issues/7947#issuecomment-495265016) This situation is undesirable because it assumes that the target platform has a bootstrapping python interpreter -available and makes the hermetic python interpreters available with `rules_python` less useful. It is also surprising to +available and makes the hermetic Python interpreters available with `rules_python` less useful. It is also surprising to users who expect bazel to output self-contained binary artifacts for a target platform. The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the python From a6fc56ed94339ad203de1a171089fe3c136c54f3 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:41:30 +1000 Subject: [PATCH 05/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 121efcf..93973c4 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -24,7 +24,7 @@ that is non-hermetic and requires a "bootstrap" python interpreter on the host. a "shebang" is always dependent on the runtime host. On Windows, there is a [native launcher](https://github.com/meteorcloudy/bazel/blob/master/src/tools/launcher/python_launcher.cc) -that launches `python.exe` on the host which then launches the `py_binary` with the same `python_stub` as macos and linux. +that launches `python.exe` on the host which then launches the `py_binary` with the same `python_stub` as macos and Linux. Related issues: * [py_binary with hermetic toolchain requires a system interpreter](https://github.com/bazelbuild/rules_python/issues/691) From a54cb0df2af9301b56101085e8f16a7f7cc5fa75 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:41:42 +1000 Subject: [PATCH 06/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 93973c4..4093b3f 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -20,7 +20,7 @@ This document describes an approach for launching `py_binary` artifacts hermetic Currently, `py_binary` is non-hermetic and launches inconsistently between platforms. On macos and Linux, there is a [python_stub](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt) -that is non-hermetic and requires a "bootstrap" python interpreter on the host. The "shebang" can be overridden, but +that is non-hermetic and requires a "bootstrap" Python interpreter on the host. The "shebang" can be overridden, but a "shebang" is always dependent on the runtime host. On Windows, there is a [native launcher](https://github.com/meteorcloudy/bazel/blob/master/src/tools/launcher/python_launcher.cc) From b3d7357c85a2eb0ab873c689a4017936ddcf55cc Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:41:54 +1000 Subject: [PATCH 07/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 4093b3f..0751a2e 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -12,7 +12,7 @@ authors: # Abstract -This document describes an approach for launching `py_binary` artifacts hermetically using the resolved python toolchain. +This document describes an approach for launching `py_binary` artifacts hermetically using the resolved Python toolchain. # Background From c50aca9bc96d8bd706d2095a9f41dab2d52a150a Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:42:09 +1000 Subject: [PATCH 08/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 0751a2e..442461f 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -32,7 +32,7 @@ Related issues: This situation is undesirable because it assumes that the target platform has a bootstrapping python interpreter available and makes the hermetic Python interpreters available with `rules_python` less useful. It is also surprising to -users who expect bazel to output self-contained binary artifacts for a target platform. +users who expect Bazel to output self-contained binary artifacts for a target platform. The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the python interpreter in the runfiles and use that to launch the program. Currently, Bazel assumes the target platform will From 91c779a25ff37942816f8e817aa28f6581898500 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:42:21 +1000 Subject: [PATCH 09/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 442461f..629d630 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -34,7 +34,7 @@ This situation is undesirable because it assumes that the target platform has a available and makes the hermetic Python interpreters available with `rules_python` less useful. It is also surprising to users who expect Bazel to output self-contained binary artifacts for a target platform. -The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the python +The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the Python interpreter in the runfiles and use that to launch the program. Currently, Bazel assumes the target platform will be able to provide the "bootstrapping" functionality. From 0d7c635d5632bde53a35343596305f05938cd1a8 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:43:19 +1000 Subject: [PATCH 10/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 629d630..d45a278 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -36,7 +36,7 @@ users who expect Bazel to output self-contained binary artifacts for a target pl The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the Python interpreter in the runfiles and use that to launch the program. Currently, Bazel assumes the target platform will -be able to provide the "bootstrapping" functionality. +be able to provide the bootstrapping functionality. # Proposal From 6c584e2f9247c8dced0ee32d18501e489cf450f9 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:43:33 +1000 Subject: [PATCH 11/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index d45a278..905bfe5 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -41,7 +41,7 @@ be able to provide the bootstrapping functionality. # Proposal -Extend the native launcher functionality to all platforms and use it to locate the relevant python interpreter and +Extend the native launcher functionality to all platforms and use it to locate the relevant Python interpreter and python program in the `runfiles` tree to launch the `py_binary`. No assumptions should be made about the target platform. In pseudo-code, the proposal is as follows: From d3cdbe836cad8eaf1b088369ddebfd2f7e3922be Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:43:46 +1000 Subject: [PATCH 12/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 905bfe5..eef0e87 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -42,7 +42,7 @@ be able to provide the bootstrapping functionality. # Proposal Extend the native launcher functionality to all platforms and use it to locate the relevant Python interpreter and -python program in the `runfiles` tree to launch the `py_binary`. No assumptions should be made about the target platform. +Python program in the `runfiles` tree to launch the `py_binary`. No assumptions should be made about the target platform. In pseudo-code, the proposal is as follows: From 2368a55506570e7911a96e415f0292278b07cfce Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:43:52 +1000 Subject: [PATCH 13/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index eef0e87..2bd4547 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -67,7 +67,7 @@ Some related work has been done that fixes Linux to Windows cross-builds of the This proposal would aim to go further and have these launchers available on all platforms, including cross_builds where appropriate toolchains are in place. Once this proposal is implemented, it would enable cross-builds of hermetic `py_binary` for all major platforms. It -would also remove the complexity introduced by having so many chains of nested execution to launch a python program. +would also remove the complexity introduced by having so many chains of nested execution to launch a Python program. Finally, while this proposal is specific to python, this solution could perhaps be reused for `java_binary`, `sh_binary` and perhaps be made available for any custom rules that require an interpreter to launch. From 72b21b533437a84cb98aa431fd47e455eceb74bb Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:43:59 +1000 Subject: [PATCH 14/16] Update designs/2022-09-12-native-launchers-python.md Co-authored-by: Richard Levasseur --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 2bd4547..ee507be 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -69,7 +69,7 @@ This proposal would aim to go further and have these launchers available on all Once this proposal is implemented, it would enable cross-builds of hermetic `py_binary` for all major platforms. It would also remove the complexity introduced by having so many chains of nested execution to launch a Python program. -Finally, while this proposal is specific to python, this solution could perhaps be reused for `java_binary`, `sh_binary` +Finally, while this proposal is specific to Python, this solution could perhaps be reused for `java_binary`, `sh_binary` and perhaps be made available for any custom rules that require an interpreter to launch. From f3ddda33bb6d03957962fbef75f688875e3301f0 Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:44:48 +1000 Subject: [PATCH 15/16] Update designs/2022-09-12-native-launchers-python.md --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index ee507be..0186a0a 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -20,7 +20,7 @@ This document describes an approach for launching `py_binary` artifacts hermetic Currently, `py_binary` is non-hermetic and launches inconsistently between platforms. On macos and Linux, there is a [python_stub](https://github.com/bazelbuild/bazel/blob/master/src/main/java/com/google/devtools/build/lib/bazel/rules/python/python_stub_template.txt) -that is non-hermetic and requires a "bootstrap" Python interpreter on the host. The "shebang" can be overridden, but +that is non-hermetic and requires a bootstrap Python interpreter on the host. The "shebang" can be overridden, but a "shebang" is always dependent on the runtime host. On Windows, there is a [native launcher](https://github.com/meteorcloudy/bazel/blob/master/src/tools/launcher/python_launcher.cc) From 2c2156ff7578e55161e1e3ccd92d8be6eac2322b Mon Sep 17 00:00:00 2001 From: Greg Roodt Date: Tue, 20 Sep 2022 21:45:15 +1000 Subject: [PATCH 16/16] Update designs/2022-09-12-native-launchers-python.md --- designs/2022-09-12-native-launchers-python.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/designs/2022-09-12-native-launchers-python.md b/designs/2022-09-12-native-launchers-python.md index 0186a0a..362a591 100644 --- a/designs/2022-09-12-native-launchers-python.md +++ b/designs/2022-09-12-native-launchers-python.md @@ -34,7 +34,7 @@ This situation is undesirable because it assumes that the target platform has a available and makes the hermetic Python interpreters available with `rules_python` less useful. It is also surprising to users who expect Bazel to output self-contained binary artifacts for a target platform. -The reason this situation exists is because of "bootstrapping". Ultimately, *something* needs to find the Python +The reason this situation exists is because of bootstrapping. Ultimately, *something* needs to find the Python interpreter in the runfiles and use that to launch the program. Currently, Bazel assumes the target platform will be able to provide the bootstrapping functionality.