Execution from source code

[expede]

A question that was raised recently was "how would one run source code directly from a Task". For example, how would you run a Python script?

We know from other systems like Nix that this is actually a nontrivial case to make reproducible. It depends on the chip architecture, OS, available system libraries, and so on.

You can specify all of these by hash ahead of time. A generic task type "run this Python" could fall into one of the following strategies:

0. If you don't care about reproducibility, set that expectation as a very loose/dangerous/attested effect
1. Fully specification the expectations of the environment (x86 Linux, with these versions down to the hash of these libraries installed)
2. Run inside of an established Docker container (by CID) that has the relevant environment set up
3. Feed the source to a Wasm Python interpreter
4. Feed the source to a Wasm compiler
5. Compile the Python to Wasm and execute it directly

2, 3, and 4 are actually really interesting, because you don't even need a new kind of Task. The source can "just" be an argument, and either use the output to further steps (4) or get the result back directly.

4 additionally gets you new Wasm that can be cached for future invocation, which is really helpful for (automatically) publishing reproducible packages to registries — which is a common workflow for e.g. a GitHub Actions.

3 and 4 and kind of microkernel-y, because you could later swap in a different (faster, bugfixed, etc) interpreter/compiler.

CC @simonwo

[simonwo]

Yes! I'm interested in a workflow that looks like "compile this {Rust/Python/Go/Zig} into WASM" and then "execute said WASM in a deterministic way", as a workflow of two separate jobs. E.g. https://docs.google.com/presentation/d/1J1M5HYpM1W64J1hCrDcsIqQZ9cTq2-a2G1a-gsY7t6g/edit#slide=id.g17a6c105ed9_0_165.

The first job is not quite as bad as "dangerously unsafe" because it at least has a consistent profile and one can apply strict memory or time limits (e.g. compile a Rust program should take <= 5 min and only communicate with crates.io). What would be cool would be to have a workflow that is "compile this program on Bac" and then "execute it on IPVM"!