Add support for Node.js without persistence

[tomayac]

No description provided.

[tomayac]

@sgbeal I managed to get a transient database to work on Node. I had to patch one self to be globalThis and it ran, see the changes (in sqlite3-node.mjs specifically) for details.

$ node demo/node.js
# Loading and initializing SQLite3 module...
# Done initializing. Running demo...
# Running SQLite3 version 3.43.2
# Creating a table...
# Insert some data using exec()...
# Query data with exec()...
# [ 20 ]
# [ 21 ]
# [ 22 ]

How can I create a persisted database, though? I tried:

// Works, but not persisted:
const db = new sqlite3.oo1.DB('./local', 'cw');

// `sqlite3.oo1.OpfsDb` doesn't exist.
const db = new sqlite3.oo1.OpfsDb('./local', 'cw');

// No such `vfs: kvvs`, same for `file:session?vfs=kvvs`. 
const db = new sqlite3.oo1.DB('file:local?vfs=kvvs', 'cw');

// Returns `0`.
sqlite3.capi.sqlite3_vfs_find("opfs")

Is this supposed to work somehow?

[sgbeal]

Bummer, i thought we had eliminated the last of the "self" references. i'll fix that when i'm back on the computer. OPFS can't work on node. i would expect dbs to be filesystem local, but what you're seeing above is the Emscripten i/o proxy, which is transient. Perhaps there's a way to tell Emscripten to use node's native i/o support?

Please excuse brevity - mobile device.

[sgbeal]

It just now occurs to me that node's native i/o is probably async, not posix-style, in which case it could not work with sqlite without an async-to-sync proxy vfs like the first OPFS vfs.

[sgbeal]

It turns out that that particular "self" reference is dead code and can be removed entirely:

Index: ext/wasm/api/sqlite3-api-worker1.js
==================================================================
--- ext/wasm/api/sqlite3-api-worker1.js
+++ ext/wasm/api/sqlite3-api-worker1.js
@@ -331,11 +331,10 @@
   'use strict';
   const toss = (...args)=>{throw new Error(args.join(' '))};
   if(!(globalThis.WorkerGlobalScope instanceof Function)){
     toss("initWorker1API() must be run from a Worker thread.");
   }
-  const self = this.self;
   const sqlite3 = this.sqlite3 || toss("Missing this.sqlite3 object.");
   const DB = sqlite3.oo1.DB;
 
   /**
      Returns the app-wide unique ID for the given db, creating one if
@@ -655,7 +654,7 @@
       //},
       result: result
     }, wState.xfer);
   };
   globalThis.postMessage({type:'sqlite3-api',result:'worker1-ready'});
-}.bind({self, sqlite3});
+}.bind({sqlite3});
 });

Upstream is being patched to remove that part and fix one more self ref which was just uncovered.

[tomayac]

Perhaps there's a way to tell Emscripten to use node's native i/o support?

@sgbeal There's the NODEFS that should work. Have you looked at this?

It uses node’s synchronous FS API to immediately persist any data written to the Emscripten file system to your local disk.

—Source

[sgbeal]

There's the NODEFS that should work. Have you looked at this?

i have not - i don't use node and don't currently have the bandwidth to go down that rabbit hole :/. Through mid-January my computing time is far more limited than usual because of familial responsibilities.

[sgbeal]

But yes, i believe doing a separate build with that enabled would resolve the persistent storage in node, and have added that to my TODOs to try out.

[tomayac]

According to the docs, it can be activated via the -lnodefs.js flag.

[zoren]

The addition of sqlite3InitModuleNode to index.mjs in itself would be helpful for creating transient databases in node.

[tomayac]

The addition of sqlite3InitModuleNode to index.mjs in itself would be helpful for creating transient databases in node.

I agree, but my fear is that people won't read the small print of "only transient databases supported" and expect full Node.js support. Hopefully we can actually make this happen soon.

[zoren]

If I use this branch I can use sqlite3InitModuleNode fine from node, but I cannot use the package from vite as it complains about the node dependencies being loaded.

[DallasHoff]

Yeah, sqlite3InitModuleNode should probably be exported from a different entry point like @sqlite.org/sqlite-wasm/node.

[tomayac]

I have just created a separate entry point for Node: node.mjs. This should allow interested parties to play with transient databases in Node, and at the same time work in the browser.

[zoren]

node needs to be before import for it to work for me:

    ".": {
      "types": "./index.d.ts",
      "node": "./node.mjs",
      "import": "./index.mjs",
      "main": "./index.mjs",
      "browser": "./index.mjs"
    },

But then it does work in both node and browser!

[sgbeal]

i feel compelled to point out that relying on object property order is not a great idea, as whether or not the order is retained is dependent on the how the object is traversed. See:

https://stackoverflow.com/a/30919039/1458521

In this case, the order of traversal is an implementation detail of whatever reads that import block, and that block may or may not use an order-honoring traversal. Since it's read as JSON, and JSON (rightfully) leaves the ordering implementation-defined, that block can get re-ordered if it passes through arbitrary JSON implementations.

[zoren]

i feel compelled to point out that relying on object property order is not a great idea, as whether or not the order is retained is dependent on the how the object is traversed. See:

I completely agree. However node does rely on the ordering of exports. For instance:

"default" This condition should always come last.

https://nodejs.org/api/packages.html#conditional-exports

[zoren]

I made a small demo repo in case anyone is interested. It's using a fork of this branch where "node": "./node.mjs", is before "import": "./index.mjs", as mentioned above.

It runs demo-123 on main thread and in a worker side by side in the browser. In node they run one after the other.

https://github.com/zoren/sqlite-wasm-demo

[tomayac]

I made a small demo repo in case anyone is interested. It's using a fork of this branch where "node": "./node.mjs", is before "import": "./index.mjs", as mentioned above.

It runs demo-123 on main thread and in a worker side by side in the browser. In node they run one after the other.

https://github.com/zoren/sqlite-wasm-demo

Thanks, merged your PR. This should allow you to use the current branch. We're slowly getting there. Transient seems solved now, pending is persistent databases.

[zoren]

Adding:

export function sqlite3InitModuleNode(opts?: InitOptions): Promise<Sqlite3Static>;

to index.d.ts would reflect the interface as it is.

But maybe it would be better if node.mjs default exported a function like index.mjs does to keep the same interface?

[PabbleDabble]

Is there value to call out that this won't work in node in a persistent context in the readme?

Our app uses WebSql (only for local development) and I'm scrambling to find a replacement after reading https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

[sgbeal]

Is there value to call out that this won't work in node in a persistent context in the readme?

Given that the only supported persistence options are browser-specific, calling out that they can't work in node seems superfluous.

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

The documentation specifically says that browsers are the only current target: https://sqlite.org/wasm/doc/trunk/about.md.

[sgbeal]

There's the NODEFS that should work. Have you looked at this?

i have not - i don't use node and don't currently have the bandwidth to go down that rabbit hole :/. Through mid-January my computing time is far more limited than usual because of familial responsibilities.

The main source tree now has a branch which modifies the node-specific build to use -lnodefs.js, which should enable NODEFS:

https://sqlite.org/src/timeline?r=wasm-nodefs&c=0bcbde7c

What is not clear is whether it will use that filesystem API by default, or whether it will still use the internal posix I/O wrappers for node. i can find no mention in the Emscripten docs about how to ensure that nodefs is made the default. It's possible that this only expose nodefs to JS, which wouldn't be helpful in our case because our module does not expose any Emscripten pieces to client-level code.

It is 100% untested as i am still, and hope to forever remain, a node-zero.

There are two options for trying this out:

First, from a checked-out copy of the main sqlite source tree:

$ fossil update wasm-nodefs
$ ./configure
$ cd ext/wasm
$ make
# go get coffee

the results will include jswasm/sqlite3-node.mjs, which uses the same jswasm/sqlite3.wasm as the rest of the builds (adding -lnodefs.js does not influence the wasm side of the build). Though i can tell emcc to build that, i've no clue how to test it.

Secondly, there is currently/temporarily a prebuilt copy of that at:

https://wasm-testing.sqlite.org/tmp/

(The filename will either be obvious or it will have been removed by the time you read this.)

[PabbleDabble]

Is there value to call out that this won't work in node in a persistent context in the readme?

Our app uses WebSql (only for local development) and I'm scrambling to find a replacement after reading https://developer.chrome.com/blog/sqlite-wasm-in-the-browser-backed-by-the-origin-private-file-system

I've spent a couple weeks now trying to get this or another option working, and just now got to this issue about node which seems to say that that this plugin doesn't support NODE + PERSTANCE

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

[sgbeal]

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

i can't say anything useful about node-based dev processes or any given JS framework (within the sqlite project we use, write, and publish only vanilla JS), but OPFS is not available in the main thread. The only main-thread persistence option is the so-called "kvvfs," but that one has severe size limitations because it uses localStorage or sessionStorage as its back-end:

https://sqlite.org/wasm/doc/trunk/persistence.md

There is unfortunately no direct path from WebSQL to OPFS because the former is main-thread-only and OPFS is worker-only. It is possible to "remote-control" an sqlite instance from the main thread via a worker-style interface, for example:

https://sqlite.org/wasm/doc/trunk/api-worker1.md

but that particular API has severe limitations (see that link) and is not recommended for anything more than very basic/toy applications.

The recommended way to operate it from the main thread, assuming that persistent storage is needed, is to move all of your db code into a worker thread and load sqlite into that same worker thread. That gives your worker full access to the API and eliminates all of the limitations of remote-controlling it over a proxy worker. The main thread would then communicate with that worker using postMessage() and high-level app-specific messages (e.g. {op:"getCustomerData",args:[1234]} instead of {sql:"select x,y,z from customerData where id=1234"). The latter is certainly possible as well, but is arguably more difficult to maintain long-term.

[thorsten-wolf-neptune]

Thank you for the response! I don't know if there's a better 'discussion' type place to ask about this, so apologies if this is off-topic. I have a ionic/cordova/angular app that I'm trying to implement a websql replacement for my local ng serve that I run in a local browswer using localhost. Should this sqlite-wasm work in a persistent manner in an angular app?

I think I got it somewhat plumbed in to a service, but I get this message Ignoring inability to install OPFS sqlite3_vfs: The OPFS sqlite3_vfs cannot run in the main thread because it requires Atomics.wait(). or this line log('OPFS is not available, created transient database', db.filename); for the other method.

@PabbleDabble We have exactely the same requirement. We offer a Low Code development platform where customers can create cordova apps and use WebSQL to store data and went now with the SQLite WASM approach as a replacement. The existing code of our customers is all in the main thread and we were challenged to get something running also for backwards compatibility. So we are using this library together with https://github.com/magieno/sqlite-client as we (and all our low code customers) cannot easily move the logic to the service worker level. Besides some performance considerations (#61) it currently works really well.

[nestarz]

I published on npm this forked version because of a critical issue with Deno (not accepting anymore the global property "self" since it's last version 1.45.3, see denoland/deno#24726):

https://www.npmjs.com/package/@eliaspourquoi/sqlite-node-wasm

It's a quick fix waiting for the owners to approve the merge of the current package so it can be used in Deno as well.

[samuelstroschein]

NodeJS is working after this #94 PR has been merged, replacing self with globalThis. I think this PR can be closed.

[tomayac]

NodeJS is working after this #94 PR has been merged, replacing self with globalThis. I think this PR can be closed.

I think most people would still want support for database persistence. I don't think this is currently possible with the official package, but I'm happy to be proven wrong. We'd need a way to implement the OPFS persistence option on the server.

[sgbeal]

I think most people would still want support for database persistence. I don't think this is currently possible with the official package,

Correct.

Not only is there a distinct lack of interest in node within the upstream project, but node already has popular SQLite wrappers, so there's no acute motivation for us to go down the rabbit hole of introducing one.

To the best of my knowledge, our experimental branch mentioned in #48 (comment) was never tested by any interested parties, so was allowed to stagnate.

We'd need a way to implement the OPFS persistence option on the server.

OPFS isn't strictly necessary - "just" a VFS which can use node's I/O API to handle the I/O on SQLite's behalf. However, like OPFS, node's I/O is very unfortunately async, which makes it exceedingly painful to integrate into an SQLite VFS.

[jlarmstrongiv]

Not only is there a distinct lack of interest in node within the upstream project, but node already has popular SQLite wrappers, so there's no acute motivation for us to go down the rabbit hole of introducing one.

While I’m glad projects like better-sqlite3 and the experimental sqlite3 module exist in NodeJS, I have found it difficult to share code between them. Each one has different implementations of everything from the db and statements, to how to exec, run, and return queries, and even whether to prefix variable names or not. I’d rather use wasm without automatic persistence and risk data loss, than deal with all of the subtle differences between libraries.

To the best of my knowledge, our experimental branch mentioned in #48 (comment) was never tested by any interested parties, so was allowed to stagnate.

I patch in that PR every time I install this package.

OPFS isn't strictly necessary - "just" a VFS which can use node's I/O API to handle the I/O on SQLite's behalf. However, like OPFS, node's I/O is very unfortunately async, which makes it exceedingly painful to integrate into an SQLite VFS.

I’d be curious how https://github.com/tndrle/node-sqlite3-wasm implements it

I’m about to go traveling, but would be glad to reply when I return. Thanks!

[joeltg]

I'm also really interested in getting this PR merged since it would let us do unit testing without launching puppeteer!

I don't think persistent databases in NodeJS are in scope for this library at all, but just loading and running transient databases in WASM is a completely reasonable expectation especially since it's so simple to support.

[tomayac]

Let's maybe do a highly scientific emoji voting:

• If you want the PR merged, knowing that it works in Node.js, but databases aren't persisted, vote with 👍.
• If you don't want the PR merged because you require persisted databases in Node.js, vote with 👎.

(As the volunteer maintainer of this package, I am slightly worried about people not reading the limitaion note that we would put up in the README and endless Issues wondering about persistence with Node.js.)

[sgbeal]

(As the volunteer maintainer of this package, I am slightly worried about people not reading the limitaion note that we would put up in the README and endless Issues wondering about persistence with Node.js.)

As the upstream maintainer i'll refrain from voting but am currently making a bag of popcorn to eat while watching this thread :).

PS: thank you, Thomas, for taking on this subproject!

[tomayac]

I can't believe the emoji reactions don't include the popcorn reaction. I had to go into DevTools to fix it. Here's my secret master plan: get hired at GitHub, add the emoji, and quit.

[joeltg]

thoughts about merging? If you add NodeJS section in the README that says "only transient databases are supported" in bold font I really don't think anybody will complain.

[sgbeal]

Speaking only for the upstream project, and not this subproject (where Thomas has full authority over what does and does not go in)...

In the upstream project we currently have no plans for supporting node-side use of the JS pieces (as distinct from node-side tools for building web apps, which we very unfortunately cannot avoid). Aside from the fact that none of us use node (so we cannot sensibly claim to support it), node already has multiple sqlite bindings and we have no desire to step on those projects' toes by releasing an "official" one.

That said: i don't mind making small tweaks which help support downstream use in node, or to adding node-specific build deliverables, but can make no commitment whatsoever as to their future stability (they may randomly break when nearby browser-targeting code is touched) nor to testing them on node.

Like Oracle databases, node is more than a framework - it's a lifestyle choice and all of my multiple-choice boxes are currently checked.

If you add NodeJS section in the README that says "only transient databases are supported" in bold font I really don't think anybody will complain.

As a counter-argument i'll point out that the upstream docs have always clearly specified that browsers are the only intended target platform, yet we still hear occasional grumbling that it doesn't work on node ;).

[tomayac]

This may be an entirely unorthodox idea, but has someone tried polyfilling the OPFS APIs on top of Node.js' fs APIs? I think there's some detection happening on the Emscripten side to detect Node.js, which could make it harder to force the package into using such a potential polyfill, as it would bail out earlier.

I quickly threw Claude at the task, and it came up with this. Someone might be able to push this further until it fulfills the relevant tests. I haven't fully thought this through, just as an idea.

[joeltg]

If there were to be a NodeJS backend it would probably be easier to make a NodeJS fs VFS implementation directly, like node-sqlite3-wasm did, than piping it through the OPFS API.

But again, persistence doesn't have to be in scope, or could at least be a separate issue.

[tomayac]

Released as 3.49.1-build2. Let's see how it goes…

[jlarmstrongiv]

Thank you @tomayac!

For continuing discussion on support for persistence in Node.js, I opened #104

I tried out the latest build—I’m glad to say it works great in Node.js, and I no longer need to patch it