Collaboration with the Loaders Team

[RaisinTen]

We had a brief discussion about how we imagine the VFS to be implemented in Node.js in the TSC meeting today - https://www.youtube.com/watch?v=SykIJ0p2Yb0. It seemed like a lot of the work the Loaders Team is doing is very close to what we are doing here regarding the VFS, so I proposed that the Single-Executable team should collaborator with the Loaders Team and @GeoffreyBooth seemed to be +1 about the idea!

The general consensus was that supporting monkey-patching is a bad idea because ESM is probably not going to support it. We should definitely find other ways.

Some context for @GeoffreyBooth, we are having some ongoing discussions in #37 regarding how we think the VFS should be implemented. Feel free to chime in there if you have opinions about any of the mentioned approaches!

Let's discuss how we can collaborate further in this thread. Should we have a cross-team meeting? Maybe a joint discussion in the Collab Summit? What does everybody think?

cc @nodejs/single-executable @nodejs/loaders

[jviotti]

Let's do it @GeoffreyBooth @RaisinTen ! At Postman, we are very interested in making this happen both in the context of SEAs and Electron. Is there a clear approach we want to go for? If so, let's start planning how we can join resources and make it happen. If not, it'd be worth arranging some discussions to figure out a solid approach.

[GeoffreyBooth]

Hey folks. In https://github.com/nodejs/loaders there have been a few side discussions of virtualized filesystems:

• Helpers design doc loaders#94 (comment)
• Proposal: Rename “loaders” loaders#95 (comment)

It’s been of interest to @arcanis and the Yarn team, as Yarn Plug-n-Play is essentially a monkey-patched fs virtual filesystem.

Monkey-patching is less possible with ESM code and with node:-prefixed modules like node:fs, and it’s always been a discouraged practice as it blurs the lines between Node internals and public, supported APIs. We would prefer to create officially supported APIs, like the Loaders API, for these use cases. nodejs/loaders#94 talks about expanding the scope of “loaders” to cover these other use cases; where “loaders” allows user customization of module loading, other API hooks would allow customization of other parts of Node like stack traces, REPL completion, filesystem calls and so on.

As far as I’m aware (@arcanis can correct me), no one’s done any design work on what such potential “customization APIs” might look like. If your team has the bandwidth to devote some time into that, I think such work would be very welcomed. We could also create a new team and repo to organize this effort, leaving “loaders” as the customization API specific to module loading and just one of several customization APIs that Node provides.

[cspotcode]

Agreed with @arcanis that those hooks are unrelated to the VFS. SEAs will be able to bake node flags, so will be able to bake node --customization-hooks my-hooks-implementation into an executable. And that'll cover all use-cases nicely.

[GeoffreyBooth]

I think it would help if someone took a stab at what a Loaders API-like API would look like for providing hooks into the filesystem. For loaders, import calls are broken into two steps, resolve and load, and we provide those two hooks to allow overriding one or both of the steps taken as part of import. I think for a theoretical Filesystem Customization API or whatever we end up calling it, we would similarly come up with a set of hooks that allow overriding key operations within the fs methods. Maybe these hooks map one-to-one with the public methods, like a readFile hook for customizing the underlying action of fs.readFile/fs.readFileSync, or maybe they don’t; someone who knows the space well should take a pass at defining what it would look like.

[RaisinTen]

I tried to come up with a draft for a Loaders API-like fs hooks API in #43.

[jviotti]

@GeoffreyBooth We also need to support child_process btw. For example, Electron supports spawning executable programs from within the ASAR by monkey-patching spawn() and friends to extract the executable program into a temporary location and executing from there.

[cspotcode]

Internal code-paths such as the following need to respect some combination of patching, VFS hooks, and/or the node:fs module override described above.
https://github.com/nodejs/node/blob/main/lib/internal/modules/esm/translators.js#L210
Otherwise modules will break when on a VFS, because certain named exports will disappear.

[jviotti]

Thanks for the links @GeoffreyBooth ! I'll paste some notes that I found interesting from them here, so the people don't have to go read the entire thing

nodejs/loaders#94 (comment)

If you really want to override all file IO, you could just replace the fs module itself via a loader. When a resolve hook sees fs or node:fs or its variations, something other than the Node builtin gets returned instead. We wouldn’t need to create distinct hooks for all the filesystem methods, I don’t think (or at least, try to get it to work with the hooks we have before proposing to add more).

This is a cool way of thinking about it. If you control the loader, you can i.e. intercept require/imports to the entire fs or child_process module and return your own. Of course, that doesn't help with the fact that you have to re-implement all the functions to work like Node.js and maintain that over time, but its an interesting way of thinking about it that we didn't consider.

nodejs/loaders#95 (comment)

Renaming the loaders repo to broaden its scope and consider use cases like fs customization sounds like a great idea. We would be happy to collaborate on this aspect there.

nodejs/loaders#95 (comment)

So in my mind, the current loaders API is really a “register these functions to alter Node’s behavior at specified points” API. And as time goes on we’ll be adding support for more named functions beyond resolve, load and globalPreload. Hence the desire for a name that better reflects what this API is.

I like this model. We should be doing this for supporting VFS'.

nodejs/loaders#95 (comment)

We could also maybe have a flag name like --register, in the spirit of the old CommonJS loaders like require('coffeescript/register') and require('@babel/register'). So --register ts-node/hooks would be like saying “register the TypeScript hooks” which corresponds with the use case, I think.

This hooks mechanism would play nice with SEAs. As I think somebody mentioned already, we can inline these arguments on the binary, and pass through that medium the implementation for the VFS, which would touch require(), import, fs and child_process.

[jviotti]

@GeoffreyBooth One interesting twist to the problem that we are considering in the context of SEAs, is how to support a VFS implementation (through hooks or monkey patching) injected on the Node.js binary itself, that would still allow native add-ons that do read operations still work OK, or at least have a way to re-write their read operations to use a native API that is VFS-aware.

[GeoffreyBooth]

Wouldn’t this work the same way as however you handle this currently via monkey-patching? Like when you monkey-patch fs.readFile or whatever, if you want certain add-ons to use the original Node readFile, I’d assume your patch needs to somehow make that determination and function as a passthrough to the original.

[cspotcode]

This question is about how sqlite.node can read a database file from the VFS, if within itself, it's doing C language FS calls? Just as an example, I don't know anything about the existence or behavior of a particular sqlite module.

[arcanis]

Whatever API is picked, native code will probably need to use something else then the stdlib (fopen & co would be difficult to make Just Work - perhaps by making node-gyp even more complicated, if anyone wants that...).

With patching, they could use the napi bindings to require fs, call its methods, and turn the resulting buffer instance into a raw data. It should already be possible, albeit slower than direct access.

Alternatively, if instead of "any VFS" we have a single format, libraries can directly include support for it in their own code. That's for example what Esbuild did, it now supports zip files natively so it doesn't need to ask JS-land to read them.

[GeoffreyBooth]

Alternatively, if instead of "any VFS" we have a single format, libraries can directly include support for it in their own code.

Can you explain this a bit more? What would this look like for Node?

[jviotti]

Whatever API is picked, native code will probably need to use something else then the stdlib (fopen & co would be difficult to make Just Work - perhaps by making node-gyp even more complicated, if anyone wants that...).

I think there are ways in which we can make this significantly inter-operable. A lot of std I/O functionality works on file descriptors. You can "open" a file on the VFS by obtaining a file descriptor to the main executable itself, seeked to the right position, and then a significant amount of I/O calls would just work (with the risk of overflowing though). You could use the napi bindings to require fs but mostly use it for fs.open.

I agree its not a 100% solution, but I feel it does reduce the scope of the problem significantly enough given my experience with Electron.