-
Notifications
You must be signed in to change notification settings - Fork 168
-
Has anyone tried to use this within a CF worker ? I think it's a great use case, put your data in some parquet files somewhere ( web server, maybe later even integrate with CF R2 ), run your query engine at the edge.
Beta Was this translation helpful? Give feedback.
All reactions
Update here:
The biggest limitation with cloudflare workers is the fact that cloudflare only implements the Service Worker API.
This hinders us to, for example, use synchronous XHRs that are a strong prerequisite for our web filesystem at the moment.
That might change in the future, but remains a blocker for the time being.
Replies: 4 comments 33 replies
-
We'd love to! But the bundle size of Cloudflare workers is unfortunately capped at 1 MB right now, which gets problematic for Wasm very quickly.
Our gzipped wasm module is around 1.8 MB so we cannot bundle the module alongside the worker script today.
(We could probably just fetch the module but we'd have to find out what that means for the startup latency of the worker)
We'd like to explore this a bit more in the following months since CF unbounded workers could be a perfect fit for certain Serverless SQL workloads.
Beta Was this translation helpful? Give feedback.
All reactions
-
Exciting! Can you please join our discord server here and write me a DM? I'd love to chat about this.
Beta Was this translation helpful? Give feedback.
All reactions
-
@ankoh You mentioned that there's a gzipped wasm module which is around 1.8 MB. I am using the one from jsdelivr and chrome dev tools say that it's 2.8 MB gzipped. Is it a different one?
Beta Was this translation helpful? Give feedback.
All reactions
-
@ilyabo brotli compression should bring you down to 1.8. maybe 2.0 in more recent builds.
Beta Was this translation helpful? Give feedback.
All reactions
-
Right, the jsdelivr ones seem to be served with gzip and are a bit larger.
Beta Was this translation helpful? Give feedback.
All reactions
-
Okay I see, thanks André
Beta Was this translation helpful? Give feedback.
All reactions
-
Update here:
The biggest limitation with cloudflare workers is the fact that cloudflare only implements the Service Worker API.
This hinders us to, for example, use synchronous XHRs that are a strong prerequisite for our web filesystem at the moment.
That might change in the future, but remains a blocker for the time being.
Beta Was this translation helpful? Give feedback.
All reactions
-
No, we do not support threads in Cloudflare Workers.
(They wouldn't be a good fit for our platform since we really want heavy compute tasks to be distributed across multiple machines. Having one Worker take multiple cores on a single machine would actually be problematic as it would make load balancing much harder. Also there are Spectre concerns.)
Beta Was this translation helpful? Give feedback.
All reactions
-
Me being lazy too, but can Workers support full-duplex requests, too? https://github.com/posener/h2conn Comes in handy to pool connections from a client if Workers was backed by a db.
Beta Was this translation helpful? Give feedback.
All reactions
-
@ignoramous haha this seems a little off topic, maybe file a feature request on workerd? :) (Actually, though, workerd itself supports full duplex even on HTTP/1.1, but the rest of the Cloudflare HTTP proxy stack does not...)
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
@kentonv thank you very much for your answer. I fully understand your policy for the production environment but how about a beta platform? 😉 I know that there are many capable duckdb persons (like @tobilg) who could help you with a beta implementation.
Beta Was this translation helpful? Give feedback.
All reactions
-
Workers really has no separate "beta platform" and if it did it would be subject to the same policy. We can explore options for implementing some form of support but the kind of multiple thread and blocking http requests discussed so far is not currently possible. Note that we are working on implementing node:fs APIs in workers using a virtual in memory file system. Would a variation on the node binding here be a possibility?
Beta Was this translation helpful? Give feedback.
All reactions
-
👍 1
-
Curious is this still in the works? Any plans to enable this?
Beta Was this translation helpful? Give feedback.
All reactions
-
@elithrar any update on this? @ankoh are u able to let us know how you were able to get this working?
Beta Was this translation helpful? Give feedback.
All reactions
-
- Is there now an async-only version of DuckDB-WASM that doesn't rely on synchronous XHR?
- Has anyone successfully used DuckDB with Durable Objects, which might provide different capabilities?
- Are there any workarounds using Web Workers or other techniques to run DuckDB in this environment?
Beta Was this translation helpful? Give feedback.
All reactions
-
Containers is out!
Beta Was this translation helpful? Give feedback.
All reactions
-
Link?
Beta Was this translation helpful? Give feedback.
All reactions
-
Beta Was this translation helpful? Give feedback.
All reactions
-
this works for duckdb?
Beta Was this translation helpful? Give feedback.
All reactions
-
this works for duckdb?
Beta Was this translation helpful? Give feedback.