🪅 Rewrite most of the toolchain logic (#3)

- Implement an unneccessarily overtuned C++ toolchain using Rust-based
uutils and the mold linker which speeds up linking by up to 95%.
- Fix hermeticity issues with system library linking. This makes the
remote execution workflow pass.
- Bundle tinfo and ncurses from Ubuntu 24.04 instead of the Nix variant.
This makes the version symbol warnings go away.
- The above also makes the `mojo` command work. That is, we now support
the Mojo REPL.
- Change the wrapping logic of the nix package. This lets us remove most
of the Bazel passthrough variables and simplifies the Bazel toolchain.

Author: aaronmondal

.github/styles/config/vocabularies/rules_mojo/accept.txt

@@ -2,3 +2,6 @@ Bazel
 [Ee]xecution
 [Tt]oolchain
 LLVM
+uutils
+patchelf
+coreutils

.github/workflows/local-remote-execution.yaml

@@ -14,7 +14,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: [ubuntu-24.04]
+        os: [large-ubuntu-22.04]
     name: ${{ matrix.os }}
     runs-on: ${{ matrix.os }}
     timeout-minutes: 45
@@ -52,4 +52,8 @@ jobs:
       - name: Run Mojo tests against the cluster
         run: |
           nix develop --impure --command \
-          bash -c "lre-bazel test --jobs=4 @mojo//... --verbose_failures"
+          bash -c "lre-bazel test \
+            --jobs=$(nproc) \
+            --keep_going \
+            --verbose_failures \
+              @mojo//..."

.gitignore

@@ -30,3 +30,7 @@ Pulumi.dev.yaml
 
 # Ignore the generated kustomization from NativeLink.
 /kustomization.yaml
+
+# These files might get created while playing around in the repository.
+hello
+mypackage.mojopkg

README.md

@@ -1,9 +1,13 @@
 # `rules_mojo` 🔥
 
-Hermetic, reproducible Bazel/Nix rules for Mojo 🔥.
+Hermetic, reproducible Bazel/Nix rules for Mojo 🔥
 
 Brings Mojo to all Linux distributions.
 
+```bash
+nix run github:TraceMachina/rules_mojo#mojo
+```
+
 ✨ **Features**:
 
 - Mojo wrapped in [Nix](https://nixos.org/). No implicit `apt` installations, no
@@ -17,6 +21,12 @@ Brings Mojo to all Linux distributions.
   which lets you execute builds in arbitrarily scalable Kubernetes clusters.
   Whether you want to run a build on 1000 cores or just share your build cache
   among friends, the NativeLink+LRE infrastructure supports it.
+- A bundled Clang/LLVM C++ toolchain with the [mold](https://github.com/rui314/mold)
+  linker and [uutils](https://github.com/uutils/coreutils), the Rust rewrite of
+  coreutils. This toolchain links Mojo executables up to two orders of magnitude
+  faster than a standard C++ toolchain.
+- Pretty compiler crash stack traces, powered by the `llvm-symbolizer` built
+  from upstream LLVM sources.
 
 🔮 **Coming soon™**:
 
@@ -31,14 +41,25 @@ Brings Mojo to all Linux distributions.
   Also, the LRE framework is still actively under development (in fact,
   `rules_mojo` acts as a validation case for LRE), so expect larger scale
   refactors.
-- At the moment there is no way to chose the Mojo version. `rules_mojo` uses
+- At the moment there is no way to choose the Mojo version. `rules_mojo` uses
   a pinned `nightly` toolchain that happened to not crash because stable
   crashed.
+- The toolchain links all executables to dynamic libraries from `nixpkgs`.
+  This brings potential performance improvements from recent `glibc` and
+  `libstdc++` versions, but makes the executables incompatible with systems that
+  don't have these libraries present. You can attempt to rewrite the `RPATH`s
+  with [patchelf](https://github.com/NixOS/patchelf), but you might run into
+  `glibc` version incompatibility issues on non-bleeding-edge systems.
+- The dynamic loader `ld-linux-x86_64.so.2` isn't set to the Nix variant.
+  This lets you run `rules_mojo`-built executables outside of the nix
+  environment. However, on older systems the version mismatch between the
+  dynamic loader and the nix-based `glibc` might lead to issues. When in doubt,
+  use `ldd <path/to/executable>` to double-check the dynamic dependency
+  resolution.
 
 🦋 **Known bugs**:
 
 - While you can build tests remotely, you need to run them locally.
-- The Mojo REPL doesn't work yet due to an `ncurses` linker symbol mismatch.
 - All mojo invocations print `crashdb` warnings because it's unclear how to
   configure the `crashdb` output location to point outside of the nix store.
 
@@ -68,13 +89,34 @@ direnv allow
 > If you don't want to use `direnv` you can use `nix develop` manually which is
 > the command that `direnv` would automatically call for you.
 
-The Mojo standard library is convenience test target for `rules_mojo`. To run
-all tests:
+Inside the Nix environment you'll have access to the `mojo` command:
+
+```bash
+# The mojo REPL.
+mojo
+
+# Build a mojo executable.
+mojo build examples/hello.mojo
+
+# Build a mojo package.
+mojo package examples/mypackage
+
+# Invoke the nix-packaged `mojo` executable
+# outside of the `rules_mojo` repository:
+nix run github:TraceMachina/rules_mojo#mojo
+```
+
+To test the `mojo_toolchain` with Bazel, run the Mojo standard library test
+suite:
 
 ```bash
 bazel test @mojo//...
 ```
 
+> [!TIP]
+> Check out [`thirdparty/mojo.BUILD.bazel`](./thirdparty/mojo.BUILD.bazel) for
+> the build file of this target.
+
 To run the example in the `examples` directory:
 
 ```bash
@@ -148,5 +190,5 @@ nix flake init -t github:TraceMachina/rules_mojo
 
 Licensed under the Apache 2.0 License with LLVM exceptions.
 
-This repository wraps the Mojo SDK which is under a proprietary license, bundled
-with the Mojo SDK and also available in the `licenses` directory.
+This repository wraps the Modular Mojo SDK which is under a proprietary license,
+bundled with the SDK and also available in the `licenses` directory.

examples/BUILD.bazel

@@ -17,6 +17,7 @@ mojo_binary(
 )
 
 mojo_test(
-    name = "hello_world_test",
+    name = "hello_test",
     srcs = ["hello.mojo"],
+    deps = [":mypackage"],
 )

examples/hello.mojo

@@ -1,5 +1,6 @@
 from mypackage.mymodule import MyPair
 
+
 fn main():
     var mine = MyPair(2, 4)
     mine.dump()

flake.nix

@@ -81,59 +81,38 @@
           , ...
           }:
           let
+            tinfo6-bin = pkgs.callPackage ./tinfo6-bin.nix { };
+
+            ncurses6-bin = pkgs.callPackage ./ncurses6-bin.nix { };
+
             native-cli = inputs.nativelink.packages.${system}.native-cli;
 
-            lre-mojo-cluster = import ./local-remote-execution/lre-mojo-cluster.nix {
+            lre-mojo-cluster = pkgs.callPackage ./local-remote-execution/lre-mojo-cluster.nix {
               inherit native-cli;
-              inherit (pkgs)
-                curl
-                git
-                kubectl
-                kustomize
-                nix
-                writeShellScriptBin;
             };
 
             lre-cc = nativelink.packages.${system}.lre-cc;
             inherit (nix2container.packages.${system}.nix2container) buildImage;
 
-            mojo-sdk = import ./mojo-sdk.nix {
-              inherit (pkgs)
-                autoPatchelfHook
-                fetchurl
-                glibc
-                icu
-                libedit
-                libgcc
-                libxml2
-                ncurses
-                stdenv
-                zlib
-                zstd;
+            mojo = pkgs.callPackage ./mojo.nix {
+              inherit (pkgs.lib) makeBinPath makeLibraryPath;
+              inherit (pkgs.llvmPackages_18) clang;
+              ncurses = ncurses6-bin;
+              tinfo = tinfo6-bin;
             };
 
-            lre-mojo = import ./local-remote-execution/lre-mojo.nix {
-              inherit (pkgs)
-                gcc
-                lib
-                ncurses
-                python312
-                zlib;
-              inherit buildImage lre-cc mojo-sdk;
+            lre-mojo = pkgs.callPackage ./local-remote-execution/lre-mojo.nix {
+              # ncurses = ncurses6-bin;
+              # tinfo = ncurses6-bin;
+              mojoEnv = self.lib.defaultMojoEnv {
+                inherit pkgs mojo;
+              };
+              inherit buildImage lre-cc;
             };
 
-            lre-kill-the-mojo = import ./local-remote-execution/lre-kill-the-mojo.nix {
-              inherit (pkgs) docker findutils kind writeShellScriptBin;
-            };
+            lre-kill-the-mojo = pkgs.callPackage ./local-remote-execution/lre-kill-the-mojo.nix { };
 
-            createWorker = import ./local-remote-execution/create-worker.nix {
-              inherit (pkgs)
-                bash
-                buildEnv
-                coreutils
-                lib
-                runCommand
-                runtimeShell;
+            createWorker = pkgs.callPackage ./local-remote-execution/create-worker.nix {
               inherit buildImage self;
               nativelink = nativelink.packages.${system}.nativelink-debug;
             };
@@ -143,11 +122,7 @@
               exec ${pkgs.bazelisk}/bin/bazelisk "$@"
             '';
 
-            lre-bazel = import ./lre-bazel.nix {
-              inherit bazel;
-              inherit (pkgs) kubectl writeShellScriptBin;
-            };
-
+            lre-bazel = pkgs.callPackage ./lre-bazel.nix { inherit bazel; };
           in
           {
             _module.args.pkgs = import self.inputs.nixpkgs {
@@ -172,21 +147,20 @@
                 LL_LDFLAGS = "-L${openssl.out}/lib";
               };
             rules_mojo.settings.mojoEnv = self.lib.defaultMojoEnv {
-              inherit pkgs mojo-sdk;
+              inherit pkgs mojo;
             };
             packages = {
-              inherit lre-mojo lre-cc mojo-sdk lre-mojo-cluster lre-kill-the-mojo bazel lre-bazel;
+              inherit lre-mojo lre-cc mojo lre-mojo-cluster lre-kill-the-mojo bazel lre-bazel;
               nativelink-worker-lre-mojo = createWorker lre-mojo;
             };
             devShells.default = pkgs.mkShell {
               nativeBuildInputs = [
-                mojo-sdk
+                mojo
                 lre-mojo-cluster
                 lre-kill-the-mojo
                 bazel
                 lre-bazel
                 pkgs.kubectl
-                pkgs.zlib
                 pkgs.python312
                 pkgs.tektoncd-cli
                 pkgs.kind

local-remote-execution/lre-mojo.nix

@@ -1,32 +1,11 @@
 { buildImage
-, gcc
-, lib
 , lre-cc
-, mojo-sdk
-, ncurses
-, python312
-, zlib
+, mojoEnv
 , ...
 }:
 
 let
-  # This environment is shared between toolchain autogen images and the final
-  # toolchain image.
-  Env = [
-    # Add all tooling here so that the generated toolchains use `/nix/store/*`
-    # paths instead of `/bin` or `/usr/bin`. This way we're guaranteed to use
-    # binary identical toolchains during local and remote execution.
-    ("PATH="
-      + (lib.strings.concatStringsSep ":" [
-      "${gcc}/bin"
-      "${mojo-sdk}/bin"
-    ]))
-    "MODULAR_HOME=${mojo-sdk}"
-    "MOJO_COMPILER=${mojo-sdk}/bin/mojo"
-    "MOJO_CC_PATH=${gcc}/bin:${mojo-sdk}/bin"
-    "MOJO_LIBRARY_PATH=${zlib}/lib:${ncurses}/lib"
-    "MOJO_PYTHON_LIBRARY=${python312}/lib"
-  ];
+  Env = mojoEnv;
 in
 buildImage {
   name = "lre-mojo";

modules/defaultMojoEnv.nix

@@ -1,12 +1,11 @@
-{ pkgs, mojo-sdk, ... }:
+{ pkgs, mojo, ... }:
 
 [
-  "MODULAR_HOME=${mojo-sdk}"
-  "MOJO_COMPILER=${mojo-sdk}/bin/mojo"
-  "MOJO_CC_PATH=${pkgs.gcc}/bin:${mojo-sdk}/bin"
-  "MOJO_LIBRARY_PATH=${pkgs.zlib}/lib:${pkgs.ncurses}/lib"
+  "MODULAR_HOME=${mojo}"
+  "MOJO_COMPILER=${mojo}/bin/mojo"
+  "MOJO_CC_PATH=${pkgs.lib.makeBinPath mojo.passthru.mojoBinPath}"
+  "LD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath mojo.passthru.mojoLibraryPath}"
 
-  # TODO(aaronmondal): This needs to be set during runtime. Let's just add it to
-  #                    a mojo wrapper script.
+  # TODO(aaronmondal): This needs to be set during runtime.
   # "MOJO_PYTHON_LIBRARY=${pkgs.python312}/lib/libpython3.12.so.1.0"
 ]

modules/rules_mojo.nix

@@ -32,14 +32,14 @@ in
 
         ```nix
         mojoEnv = [
-          "MOJO_COMPILER=''${mojo-sdk}/bin/mojo"
+          "MOJO_COMPILER=''${mojo}/bin/mojo"
         ]
         ```
 
         results in the following line in `.bazelrc.ll`:
 
         ```bash
-        build --@rules_mojo//mojo:MOJO_COMPILER=''${mojo-sdk}/bin/mojo
+        build --@rules_mojo//mojo:MOJO_COMPILER=''${mojo}/bin/mojo
         ```
 
         Supported values are: