Compilation to WASM?

[RReverser]

Miri maintainer note: this is a fun project, but not something we currently intend to support officially. To keep maintenance manageable, Miri only supports running on platforms that rustc supports running on.

Compiling the whole Rustc to WASM is a pretty big undertaking for many reasons.

However, Miri doesn't need an actual codegen and many other parts of the whole Rustc, so I wonder how realistic it would be to compile it and the pieces it depends on to WASM instead? Are there any obvious blockers?

Mostly opening this to gauge interest and estimate complexity, as I believe there is an interest in running Rust directly in the browser on playground-like websites.

P.S. Despite what I said in the first sentence, this was actually done for Clang a while ago - https://tbfleming.github.io/cib/ - which includes LLVM compiled to WASM that, in turn, generates more WASM dynamically during runtime. In theory, it should be possible to do the same for Rust, especially since they share LLVM, but for now having just an interpreter could already be an interesting starting goal.

[oli-obk]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata. That way we should be able to build a rustc which doesn't depend on llvm or other C code.

[bjorn3]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata.

I had created one in the past but it bitrotted and it was unused, so I removed it in rust-lang/rust#58847. If you copy https://github.com/bjorn3/rustc_codegen_cranelift/blob/11d816c/src/lib.rs#L182-L190 into provide and provide_extern it should work. (Also change target_features_whitelist to contain the same as cg_llvm https://github.com/rust-lang/rust/blob/2d401fb4dc89eaef5b8f31330636094f9c26b4c4/src/librustc_codegen_llvm/llvm_util.rs#L249, otherwise stdsimd wont compile.)

Another thing necessary is replacing the dlopen for loading the codegen backend with a regular extern crate. Eg replace the match at https://github.com/rust-lang/rust/blob/08bfe16129b0621bc90184f8704523d4929695ef/src/librustc_interface/util.rs#L271 with _ => || Box::new(MetadataOnlyCodegenBackend) or however you call the backend.

[bjorn3]

I want to do the same for https://github.com/bjorn3/rustc_codegen_cranelift/, but I want it to pass the rustc test suite first and cranelift doesnt support wasm output yet.

[RalfJung]

Intriguing. :) I should add one warning though: Miri isn't a fast interpreter. It's really slow. So I don't think it is actually a good environment to use to run code, I see it as more useful for debugging and testing.

But, don't let me stop you! I just felt I should give you a fair warning. And if ideas like this leak to people making Miri lightning fast while maintaining all the UB checking, I'll be even more happier. :D

[RReverser]

Miri isn't a fast interpreter. It's really slow. So I don't think it is actually a good environment to use to run code, I see it as more useful for debugging and testing.

That's understandable, but I think it's ought to be good enough for typical playground snippets :)

[RReverser]

I think a first step would be to create a new codegen backend which doesn't actually do any codegen and just dumps the metadata. That way we should be able to build a rustc which doesn't depend on llvm or other C code.

I guess that's one way, although I was wondering if Miri actually needs the main rustc crate or maybe it could be possible to depend only on some of the finer-grained rustc_* crates and avoid including codegen altogether?

[bjorn3]

The codegen backend is necessary for rustc_driver to work. Using a dummy codegen backend (<100LOC mostly copyable from the code I mentioned in #722 (comment)) is a lot easier than duplicating all the things rustc_driver does (>>1000LOC).

[RReverser]

Fair enough.

[bjorn3]

I am currently trying to compile rustc for wasm (https://github.com/bjorn3/rust/tree/compile_rustc_for_wasm), but I am hitting a compiler bug: rust-lang/rust#60540.

[RReverser]

@bjorn3 I've rebased your branch onto master, updated deps and fixed cfg's from target_env to target_os - you can check it out at https://github.com/RReverser/rust/tree/compile_rustc_for_wasm.

Eventually it compiled successfully, but then ran into the same runtime validation issue with invalid code generated by Rust.

However, I recompiled in release mode and then it passed validation!

That got me thinking it should work now, but running the generated file with wasmtime or wasmer now seems to just hang. Some infinite loop somewhere perhaps?

[RReverser]

@bjorn3 Oh... maybe it's just been taking so long (especially the compilation part). I've tried wasmer with --backend singlepass instead now, and it has actually worked!

$ ./wasmer run target/wasm32-unknown-wasi/release/rustc_binary.wasm --backend singlepass
Usage: rustc [OPTIONS] INPUT

Options:
    -h, --help          Display this message
        --cfg SPEC      Configure the compilation environment
    -L [KIND=]PATH      Add a directory to the library search path. The
                        optional KIND can be one of dependency, crate, native,
                        framework or all (the default).
    -l [KIND=]NAME      Link the generated crate(s) to the specified native
                        library NAME. The optional KIND can be one of
                        static, dylib, or framework. If omitted, dylib is
                        assumed.
        --crate-type [bin|lib|rlib|dylib|cdylib|staticlib|proc-macro]
                        Comma separated list of types of crates
                        for the compiler to emit
        --crate-name NAME
                        Specify the name of the crate being built
        --edition 2015|2018
                        Specify which edition of the compiler to use when
                        compiling code.
        --emit [asm|llvm-bc|llvm-ir|obj|metadata|link|dep-info|mir]
                        Comma separated list of types of output for the
                        compiler to emit
        --print [crate-name|file-names|sysroot|cfg|target-list|target-cpus|target-features|relocation-models|code-models|tls-models|target-spec-json|native-static-libs]
                        Comma separated list of compiler information to print
                        on stdout
    -g                  Equivalent to -C debuginfo=2
    -O                  Equivalent to -C opt-level=2
    -o FILENAME         Write output to <filename>
        --out-dir DIR   Write output to compiler-chosen filename in <dir>
        --explain OPT   Provide a detailed explanation of an error message
        --test          Build a test harness
        --target TARGET Target triple for which the code is compiled
    -W, --warn OPT      Set lint warnings
    -A, --allow OPT     Set lint allowed
    -D, --deny OPT      Set lint denied
    -F, --forbid OPT    Set lint forbidden
        --cap-lints LEVEL
                        Set the most restrictive lint level. More restrictive
                        lints are capped at this level
    -C, --codegen OPT[=VALUE]
                        Set a codegen option
    -V, --version       Print version info and exit
    -v, --verbose       Use verbose output

Additional help:
    -C help             Print codegen options
    -W help             Print 'lint' options and default settings
    -Z help             Print unstable compiler options
    --help -v           Print the full set of options rustc accepts

[bjorn3]

Oh... maybe it's just been taking so long (especially the compilation part).

Yes, it takes several minutes to compile it using wasmtime with cranelift as backend.

However, I recompiled in release mode and then it passed validation!

🎉 🎉 🎉

and it has actually worked!

I tried actually compiling something, but it errors with:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: StringError("operation not supported on wasm yet") }', src/libcore/result.rs:999:5
note: Run with `RUST_BACKTRACE=1` environment variable to display a backtrace.

I am currently trying to figure out were it errors.

[bjorn3]

Places needing patching:

• librustc_interface/util.rs: spawn_thread_pool must not spawn a thread.
• librustc/session/filesearch.rs: get_or_default_sysroot must not be called, as it needs std::env::current_exe. I am now passing an explicit --sysroot.
• librustc/session/mod.rs: build_session_ local working_dir is created using std::env::current_dir.

Edit: pushed bjorn3/rust@15f980f (based on @RReverser's branch).

Now it errors at

thread 'main' panicked at 'unknown codegen backend llvm', src/librustc_interface/util.rs:277:18

Which is expected, as I had to remove the codegen backend dynamic loader. Will try to get https://github.com/bjorn3/rustc_codegen_cranelift to work with it.

[RReverser]

Places needing patching

FWIW previously (before even filing this issue) I tried compiling rustc with Emscripten instead, which should, in theory, reduce number of these places to patch, as it supports a bit more than WASI does. Haven't gotten too far though, because I tried to build completely unpatched rustc and there were few things that still didn't compile and probably needed similar fixes as in your branch.

Which is expected, as I had to remove the codegen backend dynamic loader. Will try to get bjorn3/rustc_codegen_cranelift to work with it.

I thought the plan was to build it without any codegen, just with miri? Or do you want to build an actual full rustc?

[bjorn3]

I thought the plan was to build it without any codegen, just with miri? Or do you want to build an actual full rustc?

I want them both. :) I currently have rustc_codegen_cranelift hooked up, but rustc gives an error before rustc_codegen_cranelift can do actual codegen: can't find crate for `std` . Supporting miri will need that error to be fixed too.

[RReverser]

can't find crate for std

Yeah for that I think you'll need to do the proper build (via x.py build) to build all components. I haven't had much luck with that yet due to failures in other crates which probably also need to be patched similarly to rustc itself.

[bjorn3]

Yeah for that I think you'll need to do the proper build (via x.py build) to build all components.

Seems like it doesn't even reach the rustc version check for the libraries. I added --sysroot $(rustc --print sysroot) and disabled the rustc version check, but it still gives the same error.

[bjorn3]

Switching from wasmer to wasmtime fixed it. It even got to the beginning of codegen.

Edit: filled wasmerio/wasmer#434.

[bjorn3]

I am currently working on making miri compile for wasi, which this issue was actually about.

[bjorn3]

It seems to trap while calling the ecx.run. :(

error while processing main module ../../target/wasm32-unknown-wasi/release/rustc_binary.wasm: Instantiation error: Trap occurred while invoking start function: wasm trap at 0x2a881eb82

I pushed the wip stuff to my branch.

[RReverser]

@bjorn3 Left a comment on your MIRI commit on your branch.

[RReverser]

@bjorn3 But also, I'm not sure why rustc is now depending on miri... shouldn't it be the other way around? (like in non-WASI version)

[bjorn3]

I did that to be able to prevent having to recompile every rustc crate, which is slow and to prevent having to copy all files in the dir layout rustc wants a sysroot to be.

[RReverser]

I'm not sure I understand what you're saying... neither should be affected by which crate you compile as an entry point.

I've changed my local copy of Rust & MIRI to do just that, and got miri.wasm successfully, but yeah, also hitting some trap.

[bjorn3]

I meant that I had already compiled all crates in rust/target. When switching to miri as crate root, I would have to recompile all crates into miri/target. By keeping rustc-binary as crate root, I could reuse rust/target as target dir.

[jeff-hykin]

Got a link to the html/js for this?
This looks incredible even with all the caveats I'm sure it has!

[bjorn3]

I took https://github.com/bjorn3/browser_wasi_shim/blob/main/examples/rustc.html and made it invoke the miri I built from the second to last commit of the compile_rustc_for_wasm13 branch instead.

[jeff-hykin]

For others trying to run it, thats

• This commit: https://github.com/bjorn3/rust/tree/d8265e3fe341a6827ca97a062bcc4da6a25a4690
• And follow the instructions inside of comment.txt, but instead of the wasmtime step, take the generated dist folder and put it in the that webserver

[Nadrieril]

Somehow no one pasted this here: https://garriga.dev/rubri/

Rubri is a proof-of-concept wrapper for Miri that runs in the browser.

cc @LyonSyonII

[bjorn3]

New version available at the compile_rustc_for_wasm16 branch of https://github.com/bjorn3/rust. @oligamiq created a PR to use @whitequark's LLVM fork, so it should now be able to build for wasm, though I haven't tested linking yet. It also currently requires using wasm32-wasip1-threads, which doesn't work with browser_wasi_shim yet. You can still use the cg_clif backend with wasm32-wasip1 without threads instead by omitting --config config.llvm.toml when running ./x.py install.

[whitequark]

FWIW, I'm happy to see this effort and will support it from my side as much as I'm able to. Let me know if you need any further adjustments LLVM-side (though I'm sorta waiting on consensus on handling pthreads shims in wasi-libc at the moment).

[bjorn3]

The pthread shims thing (or adding wasip1-threads support to browser_wasi_shim) is the biggest blocker for getting this working in the browser. For complete end-to-end execution in the browser, lld would also need to be supported. I haven't checked if it already works though. Would need to rebuild LLVM again to check. And I did probably need to patch the rust standard library to support a custom host call for spawning processes such that rustc can actually invoke lld.

[oligamiq]

Regarding browser_wasi_shim's support for wasm32-wasip1-threads, it is currently under development. We're just one step away from the final stage.

rustc creates thread 1, and from there it generates 4 threads. However, after closing threads 2–5, it throws in thread 1, which causes the main wasm to not terminate. In order to terminate the currently running wasm along with its threads, the plan is to close all child threads when a throw occurs and make this observable. I plan to finish this tomorrow.

mpsc::channel is working properly. When outputting 1–1000 from both the main thread and the threads, the output alternates to some extent, so the functionality should be at a sufficient level.

However, since no thread pool has been created and files are accessible from multiple workers, the speed has significantly dropped...

I feel bad for Bjorn since I continued working after submitting the pull request when I made the file system accessible from multiple workers. Currently, I'm working on a different branch.
https://github.com/oligamiq/browser_wasi_shim/tree/wasi_multi_threads_rustc

※Require SharedArrayBuffer

[bjorn3]

I hadn't looked at your browser_wasi_shim PR yet as I assumed it to be under heavy development. Let me know when it is ready for review/ready for me to try it out.

[whitequark]

For complete end-to-end execution in the browser, lld would also need to be supported. I haven't checked if it already works though.

I've looked into that. Ideally, LLD would be built as a library and linked into the rustc binary. LLD already has a function you can call to run its main routine in-process, so you should probably use that instead of a hostcall; this has the advantage that the resulting rustc is runtime-agnostic. (I would be happy to ship yowasp-rustc for example! It would be great for firmware development.)

[bjorn3]

Ideally, LLD would be built as a library and linked into the rustc binary.

Do you have any pointers on how to do that?

I would be happy to ship yowasp-rustc for example! It would be great for firmware development.

Be aware that rustc.wasm is pretty large at 134MB. And it needs another 72MB for the standard library compiled to wasm32-wasip1-threads.

[whitequark]

Do you have any pointers on how to do that?

How familiar are you with LLVM? You're looking to call lld_main. If you can link to the right object, it seems self-explanatory after that.

Be aware that rustc.wasm is pretty large at 134MB. And it needs another 72MB for the standard library compiled to wasm32-wasip1-threads.

That sounds about right. I woulde expect LLD to add ~50M more, so ~250M before compression, maybe ~100M after. In my view this is still worth it since the object can be cached indefinitely at the client. The distribution infrastructure I'm using supports such large packages just fine.

Have you seen my FPGA toolchain integration with VS Code? The packages used in the demo are a ~50M download in total (from memory), which is a little high but is fine if you have broadband. ~100M of rustc isn't such a stretch, and I was planning on shipping yowasp-clang that would be of a similar size, anyway.

[bjorn3]

How familiar are you with LLVM? You're looking to call lld_main. If you can link to the right object, it seems self-explanatory after that.

Is enabling building of the lld standalone tool enough to build whatever static library I need to link against for lld_main to work?

[whitequark]

Is enabling building of the lld standalone tool enough to build whatever static library I need to link against for lld_main to work?

I think so, since the lld executable is a really thin wrapper (which is, iirc, autogenerated) over the function I linked to. But I would personally still scour the build system to see which target it is that does it.

[whitequark]

So after looking into it, I think you should probably build the lldWasm library and invoke lld::wasm::link, provided you can add arbitrary C++ (which IIRC you can in rustc). The lld_main I linked to earlier doesn't seem to be properly exported, and it also drags in ELF, COFF, and MachO linkers that you may not care about at all.

[bjorn3]

Managed to get linking using lld working: bjorn3/rust#8

[x0k]

Somehow no one pasted this here: https://garriga.dev/rubri/

I also use miri (and a few other solutions) here to run various programming languages in the browser: https://x0k.github.io/ppp/editor

[whitequark]

And you can use the YoWASP VS Code toolchain if you want a full editor in your browser when running programming language compilers.

[oligamiq]

https://github.com/oligamiq/rubrc
Demo project that rustc compiling code on browser.
https://oligamiq.github.io/rubrc/

@bjorn3
The embedded linker doesn't work unless it's wasm32-wasip1. Is there any way to resolve this?

Also, regarding the environment variable WASI_SYSROOT, it is required by cc, so it cannot be removed. Although the compilation can still proceed without it, unresolved imports such as __cxa_uncaught_exceptions, __cxa_decrement_exception_refcount, and __cxa_thread_atexit remain, preventing the wasm from running. For this reason, I wrote code that treats its absence as a compilation error.

[bjorn3]

The embedded linker doesn't work unless it's wasm32-wasip1. Is there any way to resolve this?

wasm32-wasip2 uses wasm-component-ld, which is a wrapper around lld. It should be possible to integrate it directly into rustc, though depending on how exactly wasm-component-ld is designed, it may require patching wasm-component-ld itself. As for native targets, it is harder than conventional cross-compilation as those generally depend on gcc or clang wrapping the linker.

[oligamiq]

For native targets, it is harder than conventional cross-compilation as those generally depend on gcc or clang wrapping the linker.

LLVM's ELF, COFF, and Mach-O linker should be included as well, but is there no way to make use of them?

[bjorn3]

You can use lld for non-wasm targets with rustc.wasm, but on Unix targets you don't normally directly invoke the linker. Instead what you invoke is gcc or clang, which in turn invoke the linker with the right arguments. Rustc doesn't (yet) support directly invoking the linker for Unix targets as it doesn't know all the arguments that are necessary to successfully link that gcc or clang would pass to the linker.

[oligamiq]

Oh, it seems I may need to pull in Clang or contribute to the Rustc code.

[oligamiq]

@bjorn3

You can use lld for non-wasm targets with rustc.wasm, but on Unix targets you don't normally directly invoke the linker. Instead what you invoke is gcc or clang, which in turn invoke the linker with the right arguments. Rustc doesn't (yet) support directly invoking the linker for Unix targets as it doesn't know all the arguments that are necessary to successfully link that gcc or clang would pass to the linker.

I changed the arguments, and successfully compiled for x86_64-unknown-linux-musl. I was also able to run the downloaded file locally without any issues.

I would appreciate it if you could kindly share the source of this information.

[bjorn3]

For musl specifically rustc has most knowledge builtin for the self-contained mode. You may be missing certain security mitigations though. For pretty much every other Unix target it doesn't have builtin knowledge.

[macovedj]

I've been using some of the rustc.wasm's that folks have produced in this issue, and wanted to ask a few questions and maybe help out if I can.

1.) Is the compile_rustc_for_wasm17 branch reliable-ish yet? I've mostly been using the compile_rustc_for_wasm16 branch because I know it works.

2.) I have access to an x86 machine, but unlimited access to an apple machine. Is there anything other than the config.toml's in these branches that is linux specific? I've modified the config and replaced an elf var with macho, and am able to produce a rustc.wasm on apple silicon, though it doesn't successfully compile rust it seems. To do so though I have to comment out this assertion as I guess the cxxflags on apple include the string we're supposed to be asserting against for some reason.

3.) I found the exchange below slightly confusing:

The embedded linker doesn't work unless it's wasm32-wasip1. Is there any way to resolve this?

wasm32-wasip2 uses wasm-component-ld, which is a wrapper around lld. It should be possible to integrate it directly into rustc, though depending on how exactly wasm-component-ld is designed, it may require patching wasm-component-ld itself. As for native targets, it is harder than conventional cross-compilation as those generally depend on gcc or clang wrapping the linker.

It seems to indicate that you've used the linker when targeting wasm32-wasip1 @oligamiq?, but also @bjorn3 and the comment.txt files seem to indicate that that isn't supported? For my use case, I'm primarily interested in targeting wasm32-wasip1 rather than wasm32-wasip2, but want to make heavy use of linking. Given that, is it expected that in it's current state, the latest rustc.wasm can be used with lld to link crates when targeting p1? Or would that require building with wasm-ld instead of lld since I'm targeting p1? I've tried using rustc.wasm with the -l flag and end up getting the following panic

thread 'main' panicked at compiler/rustc_session/src/filesearch.rs:199:62:
Failed finding sysroot: "current_dll_path not supported"

Thanks!

[bjorn3]

Is the compile_rustc_for_wasm17 branch reliable-ish yet? I've mostly been using the compile_rustc_for_wasm16 branch because I know it works.

If I recall correctly there is not much difference between the two other than a rebase. I currently use compile_rustc_for_wasm17 as scratchpad that I force-push to occasionally until there are significant changes to announce here, at which point I switch to working on a new branch.

2.) I have access to an x86 machine, but unlimited access to an apple machine. Is there anything other than the config.toml's in these branches that is linux specific? I've modified the config and replaced an elf var with macho, and am able to produce a rustc.wasm on apple silicon, though it doesn't successfully compile rust it seems. To do so though I have to comment out this assertion as I guess the cxxflags on apple include the string we're supposed to be asserting against for some reason.

I don't think there is anything linux specific, but it's been a while since I last looked.

I found the exchange below slightly confusing:
[...]
It seems to indicate that you've used the linker when targeting wasm32-wasip1 @oligamiq?, but also @bjorn3 and the comment.txt files seem to indicate that that isn't supported? For my use case, I'm primarily interested in targeting wasm32-wasip1 rather than wasm32-wasip2, but want to make heavy use of linking. Given that, is it expected that in it's current state, the latest rustc.wasm can be used with lld to link crates when targeting p1? Or would that require building with wasm-ld instead of lld since I'm targeting p1? I've tried using rustc.wasm with the -l flag and end up getting the following panic

This exchange was about using the produced rustc.wasm to compile binaries for wasm. rustc.wasm itself needs to be compiled for wasm32-wasip1-threads for the time being. (It might start to work with minor changes on wasm32-wasip1 with the next wasi-sdk, but that depends on if the emulated C++ mutexes for single threaded wasm land before wasi-sdk does a new release or not.)

[oligamiq]

1.) Is the compile_rustc_for_wasm17 branch reliable-ish yet? I've mostly been using the compile_rustc_for_wasm16 branch because I know it works.

As far as I know, it was broken due to the rustc update two weeks ago.

2.) I have access to an x86 machine, but unlimited access to an apple machine. Is there anything other than the config.toml's in these branches that is linux specific? I've modified the config and replaced an elf var with macho, and am able to produce a rustc.wasm on apple silicon, though it doesn't successfully compile rust it seems. To do so though I have to comment out this assertion as I guess the cxxflags on apple include the string we're supposed to be asserting against for some reason.

I’m not familiar with macOS, so I don’t know.

3.) I found the exchange below slightly confusing:
[...]
It seems to indicate that you've used the linker when targeting wasm32-wasip1 @oligamiq?, but also @bjorn3 and the comment.txt files seem to indicate that that isn't supported? For my use case, I'm primarily interested in targeting wasm32-wasip1 rather than wasm32-wasip2, but want to make heavy use of linking. Given that, is it expected that in it's current state, the latest rustc.wasm can be used with lld to link crates when targeting p1?

In compile_rustc_for_wasm16, it should have been possible to compile directly to .wasm using the wasm-ld that was included by default, but the arguments might have been incorrect.

This is an example of the compile command:
rustc /main.rs --sysroot /sysroot --target wasm32-wasip1 --out-dir /tmp.

Would that require building with wasm-ld instead of lld since I'm targeting p1? I've tried using rustc.wasm with the -l flag and end up getting the following panic

thread 'main' panicked at compiler/rustc_session/src/filesearch.rs:199:62:
Failed finding sysroot: "current_dll_path not supported"

rustc doesn't properly recognize the --sysroot option. For example, even if you add the --sysroot argument to rustc -vV, it is not used. This causes the error. Currently, there is no workaround, and you will need to force an overwrite as shown in https://github.com/oligamiq/rust/commit/78681c46948a62f2d8abca7611fa1ae071d85d40#diff-15fb047830d3dd795e873a797845f3b7769cec990c00ec1f1ec4d1347cbf1033