feat: split bundler-friendly bundle in to individual VFSs to reduce import size - #168

As we're at it, Emscripten just added support for the proposed Cross-Origin Storage (COS) API. I'd love if you could see if there's a way to (i) compile with the new flag and (ii) see if the JS bundles could be optimized for COS. This would be a pretty big lever given SQLite Wasm is used on many sites. Full disclosure: I'm on the team that wants to make COS happen in browsers. Having SQLite as a project supporter would be huge. Are you interested in looking into it, @jurerotar and @sgbeal.

COS sounds awesome! Count me in, I'd happy to help with this!

I created a test build with the cross-origin storage enabled, seems to (at least) build with no issues 😄

https://github.com/jurerotar/sqlite-wasm-emcc-cos

I created a test build with the cross-origin storage enabled, seems to (at least) build with no issues 😄

https://github.com/jurerotar/sqlite-wasm-emcc-cos

Amazing! The glue code might need some changes. If you install the COS extension and open a SQLite demo page using this build, you should see the Wasm binary be used (in DevTools, it loads a blob: URL) when it works correctly, but I think the glue code has custom loading.

Yup, you're right, getting no resources found :(
We'll need some upstream changes then.

I hacked something together in https://googlechrome.github.io/samples/sqlite-wasm-cos/. Maybe you can extrapolate from there? I don't think this would work for all variants of the various scripts yet.

Alrighty, as you correctly assessed, SQLite-WASM uses a custom loader in the form of Module.instantiateWasm. I've tried monkey-patching said loader and it seems to work. It's not the prettiest solution, but it gets us there with least amount of breaking changes.

@sgbeal, I know you haven't said yes to this work, but if I'm operating under the assumption that you will: do we still require Module.instantiateWasm to load WASM? If not, Emscripten provides their own implementation of COS-compliant file loading. If yes, I can post a (so far) very crude monkey-patch we can add.

i've not looked closely at any of this because (A) i'm nursing a newly-operated dog and (B) it's so hot there that my computer is off most of the time, but FWIW...

i fully sympathize with the complaint and am not against, e.g., adding flags to the upstream build to filter select pieces out at build-time. The infrastructure is all there to support it. It's likely to be at least 2-3 weeks before i can explore that but am flagging this email thread so it (削除) won't (削除ここまで) is far less likely to get lost.

COS i'm hearing of for the first time. The link says:

This feature is experimental. The underlying Cross-Origin Storage browser API is a WICG proposal that has not yet shipped in any browser.

Until it's in one of the Big Three browsers and in progress in the other two, we're not likely to look twice at it upstream.

COS i'm hearing of for the first time. The link says:

This feature is experimental. The underlying Cross-Origin Storage browser API is a WICG proposal that has not yet shipped in any browser.

Until it's in one of the Big Three browsers and in progress in the other two, we're not likely to look twice at it upstream.

It's no wonder that you hear about this the first time, it's relatively new. We have an extension that you can install to benefit from it today; it's a progressive enhancement. The chicken and egg problem for making it happen in ideally all browsers is to show developer demand, so if we can say SQLite uses this (just like Transformers.js, WebLLM, wllama, Flutter, and Emscripten today already*), this would be a huge lever. The cache benefits for a widely deployed resource like SQLite Wasm are real, and I'd love for you to take part in this.

(Hope the dog is fine! ❤️🐶)

——

• https://chromestatus.com/feature/5163371507875840#:~:text=Web%20/%20Framework%20developer%20views%20notes

I hope your dog has a speedy and painless recovery, Stephan!

My solution for splitting modules is pretty brittle, because I have to split already generated code (https://github.com/jurerotar/sqlite-wasm/blob/a99c7de5234cbb57e846b1dac574c2c09ba7bc18/scripts/split-bundler-friendly-vfss.mjs)

Stephan, if you're okay with it, I'd be happy to prepare an example PR on how we could implement the changes that I did in this PR, upstream. I would of course focus on backwards compatibility, so existing consumers wouldn't notice any difference. Main changes would be introducing a module system, so treeshakability becomes built-in at the source. If you then don't like my solution, we can think of an alternative one, or stick with this PR.

Please let me know if this is something you'd like to see from my side! 😄

The chicken and egg problem for making it happen in ideally all browsers is to show developer demand, so if we can say SQLite uses this (just like Transformers.js, WebLLM, wllama, Flutter, and Emscripten today already*), this would be a huge lever.

A long list of long-term commitments and responsibilities forces me to be highly selective, and features which aren't imminent in all of the major browsers are among the first to get filtered out as candidate side-projects.

i won't quite say "never" but will point to the recent OPFS Web Locks addition as an example: Web Locks was never seriously considered as an option until it was in all of the Big Three Browsers.

(Hope the dog is fine! ❤️🐶)

She's recovering but it will be about two weeks before she can take the stairs and hop on and off the bed (until then i'm sleeping on the floor so that she won't rip her stitches trying to get into the bed). Sigh. #Parenthood

Jure:

We already have some pieces in place anticipating this capability but it's never been fleshed out. It's really just a matter of figuring out how it should plug it into the build process. This is just spitballing, but maybe just something like:

$ make opfs=0 opfs-ws=1 kvvfs=0
# or maybe:
$ make options="opfs-ws"

defaulting to everything on for compatibility. The resulting sqlite3.mjs would simply be missing any of the disabled options.

:-?

At Chrome, we're confident enough to put COS in Emscripten. What if Jure were to remodel the Wasm instantiation logic so it stays closer to Emscripten's (i.e., reducing the code you as SQLite are responsible for), would this be an option?

Literally LOL'ing about your dog story. Hope you at least have a good air matress!

What if Jure were to...

It's still nowhere on my radar for the time being. My local bandwidth simply doesn't support nice-to-haves right now. This filtering of the build is arguably also nice-to-have, but it's one which has bugged me for a couple of years and some work has already been done in that direction so it won't be a big effort to get that working. COS, on the other hand, lives in a whole other scope. Maybe someday, but not today.

Hope you at least have a good air matress!

i'd completely forgotten about my air mattress, but it's too big for the current room arrangements :/. It was bought when i had to wait a few weeks for my furniture after moving in.

BTW:

Rough comparison is here. We save roughly 150-200KB, depending on VFS used, which is around 50KB gzip on average. To be 100% honest, I was going for a bit larger numbers, but core is mostly Emscripten code and there's not much I can do about it.

Search the makefile for "bare-bones". That removes all of the "optional" stuff from the WASM file but does not (AFAIR) affect the JS (which instead (currently) loads everything but skips adding bindings for optional C APIs which aren't in the WASM exports). Like you, we were underwhelmed at the amount of real savings so have never really explored a more modular build:

$ make oz barebones=1 # oz=optimize for smaller size, barebones=1 omits a bunch of C APIs
...
$ ls -la jswasm/*.{mjs,wasm}
-rw-rw-r-- 1 stephan stephan 740572 Jun 23 22:05 jswasm/speedtest1.wasm
-rw-rw-r-- 1 stephan stephan 788310 Jun 23 22:06 jswasm/sqlite3-64bit.mjs
-rw-rw-r-- 1 stephan stephan 626136 Jun 23 22:05 jswasm/sqlite3-64bit.wasm
-rw-rw-r-- 1 stephan stephan 785423 Jun 23 22:06 jswasm/sqlite3.mjs
-rw-rw-r-- 1 stephan stephan 614096 Jun 23 22:05 jswasm/sqlite3.wasm
-r--r--r-- 1 stephan stephan 1516 Jun 23 22:05 jswasm/sqlite3-worker1-bundler-friendly.mjs
-r--r--r-- 1 stephan stephan 1499 Jun 23 22:05 jswasm/sqlite3-worker1.mjs
-r--r--r-- 1 stephan stephan 12692 Jun 23 22:05 jswasm/sqlite3-worker1-promiser-bundler-friendly.mjs
-r--r--r-- 1 stephan stephan 12675 Jun 23 22:05 jswasm/sqlite3-worker1-promiser.mjs
# mjs stripped of comments:
$ stripccomments < jswasm/sqlite3.mjs | wc
 13883 33926 414149

Upstream we now have an experimental branch:

https://sqlite.org/src/timeline?r=js-omit-pieces

which allows us to do:

$ make clean; make omit-api="kvvfs OPFS"; # OPFS=>all OPFS vfses
$ make clean; make omit-api="opfs opfs-wl"; # retains opfs-sahpool

The feature/API names are: opfs opfs-sahpool opfs-wl kvvfs oo1 vtab

(You don't want to disable oo1, as that's the API 99% of users (and the test suite) are using, but the option is there for those who really do want it, and more options may be added as good candidates are found.)

This can be used together with the npm target:

$ make npm omit-api="kvvfs"

However, the resulting file names are not unique to each build, so a small amount of scripting would be involved in getting multiple distinct builds going.

Jure, would that get you where you want to go? Please suggest any changes. If you have a concrete set of builds in mind we can add the aforementioned small amount of scripting in the makefile to generate whatever combinations you'd like to host.

You beat me to it, Stephan! 😄

I took some time and prepared how I envisioned this to work. A preview is available here: https://github.com/jurerotar/sqlite-modular-wasm/pull/1/changes. Keep in mind, it's very much in a first draft state.

The idea behind it was to move away from globalThis.sqlite3ApiBootstrap.initializers. This is what essentially prevents bundlers from tree-shaking the unused modules. I've instead reworked the individual .js files to behave like modules. Each one only imports and exports stuff, but doesn't add anything to global namespace.

Backwards compatibility is then easily ensured by just creating an additional module, that imports all individual modules and registers them to the global namespace.

As a result, generated bundle is also much smaller. Previously, bundler-friendly, node, worker-1 scripts all generated full implementations (around 24k LoC each) of everything. Now, each of these files are just Emscripten code + individual imports.

Here's how to output looks like:

If you find the time, please take a look and let me know what you think of this approach! 😄

The idea behind it was to move away from globalThis.sqlite3ApiBootstrap.initializers.

That is unlikely to happen upstream. Those functions are a necessary level of indirection for the various build modes we support and central to our ability to plug the sub-APIs in and out (plus it gives downstream folks a handy way to plug pieces in to their custom builds).

I've instead reworked the individual .js files to behave like modules. Each one only imports and exports stuff, but doesn't add anything to global namespace.

Upstream has to support vanilla JS as well, as well as communicate certain state across boundaries which Emscripten walls off, making the global scope a necessary evil (but a temporary one - the library removes all of that during its bootstrapping). Had it been developed about 3 years later we might have skipped vanilla support altogether, but we're stuck with it forever. Had i known at the time that import() can be used like a function to pull ESM modules into non-ESM code, we might have targeted only ESM, too. Mistakes were made and now i've got to support them forever (because that's the project policy upstream).

We could maybe factor out the initializer queue for (just) ESM builds, but that would be a highly disruptive change i've not got the motivation to pursue.

Stephan, thank you for the explanation!

I think your solution would work best then. On our side, we'll just need to make sure to run both the "regular" npm build (to continue supporting existing API) and the "specific" ones.

However, the resulting file names are not unique to each build, so a small amount of scripting would be involved in getting multiple distinct builds going.

This should be fine to handle on our side. Let me see if I can get it running today and I'll update you on my progress.

Thank you for the help!

With apologies for the delayed response - it's simply been too hot to be on the computer (until about an hour ago).

Stephan, there's a consistent 20KB difference. The reason for it is worker1 API being included in your builds. Any chance it could be omitted from the the omit builds?

We can add another omit for that, yes. i'll try to get to that today.

Minor caveat: none of this changes how the WASM file is built. Thus even if kvvfs (for example) is disabled, its native side is getting built in. kvvfs is a hybrid VFS: it's implemented in native code to work with local files and the JS pieces swap out some of the VFS methods to redirect where the key/value storage is. Disabling all optional stuff in the WASM file saves, IIRC, only 200kb (try "make clean; make barebones=1").

We can add another omit for that, yes. i'll try to get to that today.

Support for omit-api=worker1 and omit-api=ALL were just added to the experimental branch. That will get merged into trunk once we've validated that does what you're looking to do.

PS: don't use "omit-api=ALL" for your builds because it will remove oo1, which probably 99.9% of users of this lib are using. That option exists for the sake of those looking to add a custom OO wrapper on top of the C-level APIs.

With apologies for the delayed response - it's simply been too hot to be on the computer (until about an hour ago).

Completely understandable!

Minor caveat: none of this changes how the WASM file is built.

Yup, makes sense! Thanks for the heads up!

On a separate note, I've been digging a little bit deeper, and there's one more thing we can potentially omit. Would it be possible to omit the capi and the code needed to get it working? The idea here being that some users may not need capi if they're happy with oo1 and what it provides.

Ideally, this would still work through a plugin-like system (import only the functionalities you need), but I'm aware of the restrictions we have.

On a separate note, I've been digging a little bit deeper, and there's one more thing we can potentially omit. Would it be possible to omit the capi and the code needed to get it working? The idea here being that some users may not need capi if they're happy with oo1 and what it provides.

oo1 is just a thin wrapper around capi, which sits one level above the raw WASM exports. capi does all of the argument and result type conversion to/from WASM/JS. If we remove that, the only thing left is the raw WASM exports, which are very nearly useless without a higher-level argument/result conversion framework.

Ah, thank you for the explanation. I think we should be good then with the omit solution. I'll update this PR later today. Thank you for the help!

BTW: this is in trunk now.

Also BTW: i've been thinking about "omit-capi" and will look into that option at some point. i believe that all we need to do is omit essentially everything except the emscripten-injected pieces and (maybe) the initialization/bootstrapping infrastructure. That would provide an sqlite3.wasm and its requires Emscripten-provided imports, but users would be on their own as far as actually making any use of it. They'd need to provide their own replacement for the JS/WASM function args/results conversions (Emscripten's are not flexible enough for more than very basic use cases, though, and Emscripten cannot nicely bridge C structs between WASM and JS). Not impossible, by any means, but a boatload of work is involved so the real audience for such a thing is probably very close to zero.

Is this still being actively pursued?