Support language agnostic wit worlds

[MendyBerger]

Currently the world being targeted from Go has to explicitly include wasi:cli/imports@0.2.0;.
This is not a big deal when building a special world for Go, but becomes a problem when generating bindings to a world not designed with Go in mind.
It would be great it the Go tooling would match other language's tooling, and wouldn't require this include.

[ydnar]

This is a limitation of the Go runtime, which has expectations for certain APIs present in the host environment.

You can make a Go-only world that includes your other language world.

[MendyBerger]

Thanks for the quick response @ydnar!

I don't think that this should require the imported world to include wasi:cli explicitly. The module can expect multiple worlds, and one can just be the internal wasi:cli/imports.

At the very least, this is possible to do in Rust. Happy to throw together an example if you'd find it helpful.

[ydnar]

The fact that the WIT files are needed post compilation is an anti-pattern.

We have WIP code to synthesize narrowly tailored Component Model metadata for each Go package, along with WIP code to import this metadata into TinyGo during the build phase.

This would allow developers to skip the wasm-tools component embed step, and simplify the wasm-tools component new phase by embedding all necessary WIT in the output binary from TinyGo.

Second, I hope to lean on work by Luke Wagner et al to enable optional imports, so the stdlib can optionally depend on a reasonable subset of the CLI world.

[lukewagner]

Agreed with @MendyBerger that it should be possible (for any language) to compile components that target arbitrary worlds (such as, in the limit, worlds with zero imports). This supports one of wasm's main use cases as a plugin where custom execution environments (and thus worlds) are the norm.

But also agreed with @ydnar that we should address this use case without requiring major changes to the overall structure of existing languages' toolchains, which all tend to assume an ambient wasi:filesystem, among other things.

In particular, the implementation constraint I've seen in not just Go, but most languages' standard libraries is that the standard library implementation is written and compiled once, monolithically, well before any per-component "target world" is known. While we could imagine various sophisticated compilation schemes that would optimally strip or virtualize different parts of the standard library based on the target world, these are more intrusive and so probably a non-starter (at least initially).

Instead, the lowest-impact, easiest-to-understand approach we've found to reconcile these competing requirements is the "optional imports" approach that @ydnar mentioned, which I'd summarize as:

• Well-behaved language compilers/runtimes/standard-libraries should "optionally" import everything.
• All functions that are "optionally" imported can be imported as usual (with the same core wasm signatures), but they are well-defined to trap if the host doesn't actually provide the import. To avoid this trap, core wasm must import and query an additional per-optional-import "feature test" function that, in core wasm, has a well-defined mangled import name.
• Thus, a well-behaved language compiler/runtime/standard-library should guard usage of optionally-imported functions by first calling the feature-test import and handling the "no" case gracefully (by synthesizing appropriate errors, perhaps virtualizing something in-memory, or something else).

Thus, from a core wasm toolchain's perspective, this works a lot like how feature testing works on the Web in JS. (The main difference is that the features being dynamically tested for are declared statically as imports.)

Eventually, the idea is to support optionality directly in WIT/C-M by adding an optional type constructor which would allow you to write, e.g., import optional wasi:filesystem/types in a world. But to support the basic use case described in the root comment, I don't think we actually depend on touching WIT/C-M; we can achieve the goal with a few tactical additions to the toolchain:

• In the Go bindings generator, we could add a flag saying "make these (or all) interfaces optional", which would then emit the additional feature-test functions.
• In wasm-component-ld, which does know the desired target world, if a core import in the given core module is observed that is not present in the target world but is optionally imported (i.e., the feature-test function is imported), wasm-component-ld can trivially implement the feature-test function with a constant-valued "false" function and every other function with unreachable.

If someone is keen to work on this to fix this problem, I think they could get started now, so let me know if anyone is.