Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Cloudflare workers #430

Answered by ankoh
dantodor asked this question in Q&A
Nov 27, 2021 · 4 comments · 33 replies
Discussion options

Has anyone tried to use this within a CF worker ? I think it's a great use case, put your data in some parquet files somewhere ( web server, maybe later even integrate with CF R2 ), run your query engine at the edge.

You must be logged in to vote

Update here:
The biggest limitation with cloudflare workers is the fact that cloudflare only implements the Service Worker API.
This hinders us to, for example, use synchronous XHRs that are a strong prerequisite for our web filesystem at the moment.
That might change in the future, but remains a blocker for the time being.

Replies: 4 comments 33 replies

Comment options

We'd love to! But the bundle size of Cloudflare workers is unfortunately capped at 1 MB right now, which gets problematic for Wasm very quickly.
Our gzipped wasm module is around 1.8 MB so we cannot bundle the module alongside the worker script today.
(We could probably just fetch the module but we'd have to find out what that means for the startup latency of the worker)
We'd like to explore this a bit more in the following months since CF unbounded workers could be a perfect fit for certain Serverless SQL workloads.

You must be logged in to vote
6 replies
Comment options

ankoh Jan 11, 2022
Collaborator

Exciting! Can you please join our discord server here and write me a DM? I'd love to chat about this.

Comment options

@ankoh You mentioned that there's a gzipped wasm module which is around 1.8 MB. I am using the one from jsdelivr and chrome dev tools say that it's 2.8 MB gzipped. Is it a different one?

image

Comment options

ankoh Jan 17, 2022
Collaborator

@ilyabo brotli compression should bring you down to 1.8. maybe 2.0 in more recent builds.

Comment options

ankoh Jan 17, 2022
Collaborator

Right, the jsdelivr ones seem to be served with gzip and are a bit larger.

Comment options

Okay I see, thanks André

Comment options

Update here:
The biggest limitation with cloudflare workers is the fact that cloudflare only implements the Service Worker API.
This hinders us to, for example, use synchronous XHRs that are a strong prerequisite for our web filesystem at the moment.
That might change in the future, but remains a blocker for the time being.

You must be logged in to vote
19 replies
Comment options

No, we do not support threads in Cloudflare Workers.

(They wouldn't be a good fit for our platform since we really want heavy compute tasks to be distributed across multiple machines. Having one Worker take multiple cores on a single machine would actually be problematic as it would make load balancing much harder. Also there are Spectre concerns.)

Comment options

Me being lazy too, but can Workers support full-duplex requests, too? https://github.com/posener/h2conn Comes in handy to pool connections from a client if Workers was backed by a db.

Comment options

@ignoramous haha this seems a little off topic, maybe file a feature request on workerd? :) (Actually, though, workerd itself supports full duplex even on HTTP/1.1, but the rest of the Cloudflare HTTP proxy stack does not...)

Comment options

@kentonv thank you very much for your answer. I fully understand your policy for the production environment but how about a beta platform? 😉 I know that there are many capable duckdb persons (like @tobilg) who could help you with a beta implementation.

Comment options

Workers really has no separate "beta platform" and if it did it would be subject to the same policy. We can explore options for implementing some form of support but the kind of multiple thread and blocking http requests discussed so far is not currently possible. Note that we are working on implementing node:fs APIs in workers using a virtual in memory file system. Would a variation on the node binding here be a possibility?

Answer selected by ankoh
Comment options

Curious is this still in the works? Any plans to enable this?

You must be logged in to vote
1 reply
Comment options

@elithrar any update on this? @ankoh are u able to let us know how you were able to get this working?

Comment options

Folks @ankoh / @elithrar

  1. Is there now an async-only version of DuckDB-WASM that doesn't rely on synchronous XHR?
  2. Has anyone successfully used DuckDB with Durable Objects, which might provide different capabilities?
  3. Are there any workarounds using Web Workers or other techniques to run DuckDB in this environment?
You must be logged in to vote
7 replies
Comment options

Containers is out!

Comment options

Link?

Comment options

Comment options

this works for duckdb?

Comment options

this works for duckdb?

See https://github.com/tobilg/cloudflare-duckdb

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

AltStyle によって変換されたページ (->オリジナル) /