If worker.ready is just a reference to a promise that resolves when stuff is, er, "ready", we should update the docs.

I've been thinking about @mchilvers use of workers via the script tag:

<script type="mpy" src="main.py" async></script>
<script type="py" src="heavy_stuff.py" worker></script>

I imagine this "feels" quite nice from the perspective that there's only one place (in the HTML) where all the Python code is started.

I wonder if we give a worker tag an id:

<script type="py" src="heavy_stuff.py" worker id="fred"></script>

Then perhaps we should be able to easily grab a reference to the resulting PyWorker instance from somewhere? Then I'd be able to do something like this in the main thread:

from pyscript import active_workers
fred = active_workers["fred"]
await fred.ready
... do stuff ...

The name active_workers is just a placeholder (better naming suggestions welcome), and only worker tags with a valid id get referenced in there. It's just a Python dict.

Thoughts..?

@mchilvers, does this help address your "beginner's eye view of workers" we talked about? @WebReflection can you see any gotchas or blind spots?

To be clear, this isn't a new feature, but a convenience for folks who just want to kick off workers via a script tag.

an id can be used already because of the xworker exposed by the script ... the current solution within Python code is better and more elegant, the rest feels like a slippery slope to me ...

• you can't know when the MutationObserver reached and bootstrapped that script so it's surprise prone if no xworker is attached (which is a polyscript feature, not a PyScript one)
• runtime scripts won't be on the page or easily reachable via ID
• a script can exist with or without a worker attribute ... they are all independent
• the HTML is not the source of truth because runtime scripts can be created anyway
• with explicit PyWorker creation within your code you have full ownership of that worker ... with a shared script on the page you end up with possible sync name clashing conflicts and who knows what else if two scripts deal with the same shared xworker ...

I mean, it's aloready possible to shoot oneself' feet via hidden polyscript features and I am not super happy in promoting such practice in the wild, it's just more problematic for very little benefit.

Also, a worker has no guarantees that other scripts bootstrapped and provided sync features (in case these need features from the main thread) so this anti-correlational pattern really screams for troubles.

If you bootstrap your PyWorker you have full control and nothing else can mess up with that.

If you do want stuff to mess up though, you add a promise to the window that resolve with the xworker once your script landed and it's bootstrapped on the page so that any other script can reach it ... this is also known as SharedWorker but it's a way cleaner API and it doesn't need HTML at all as it can work even within workers (pyodide worker using a shared pyodide worker or mpy worker using a shared pyodide worker and so on) so that if this is what we really want or need (I still need to see a use case for that) I'd rather provide a SharedPyWorker API still coming from the stdlib, not augmented without control on the HTML page.

HTML can also stream so there's zero guarantee the ID is reachable when a mpy script started ... I think we are better off explaining how to bootstrap a Pyodide worker in our docs ... everything else seems to me looking for troubles.

P.S. that being said ... because the xworker.ready is already there and it's the solution, one can use an id, grab that script, hope for the best that xworker exists and it's been botstrapped already, await ready and use its exposed methods ... nothing to do from our side, still we need to explain that xworker is attached when the MutationObserver find that node and bootstraps it, never before, like it is for Custom Elements that work only after their definition lands on the registry.

99% of the time this won't be an issue, but in bigger projects/pages this might be an issue, so that at least that shared worker should be before anything else to be sure it's the first that our MutationObaserver bootstrap ... this is all about documentation though, nothing really to change in our code .... the id to reach a node is a well known HTML feature, nothing there for us to change neither.

OK... Thanks for all the clarifications. I think the TL;DR here is we should be VERY CLEAR in the docs about how to start workers. Furthermore, @mchilvers's approach of using a script to start a worker should be discouraged because of all the reasons you state. This is very much a docs/communication/idioms problem because I don't think Martin was unreasonable in his assumption that "it should just work that way". I.e. we need to be explicit that it's not a good idea.

Looks like the worker pages in the docs will need some love to address this.

I don't think Martin was unreasonable in his assumption that "it should just work that way"

neither do I ... it's just lack of documentation how to do that ... I've missed it during the docs week but to clarify, Martin wasn't initializing a worker, any worker as script initializes itself, Martin was exploiting a Polyscript property needed for JS only purposes. The fact he got that exploit means he also read docs (and I don't even know where) but that's not how PyScript should offer, imho.

I've tried to say that already during the meeting but after re-watching it I think I wasn't clear, I was also slow in reading, understanding, and providing proper explanations without correcting myself as I really couldn't understand why anyone would end up doing an addEventListener dance to bootstrap a Pyodide worker because to date nobody ever tried that, not even via JS, as the PyWorker and MPWorker are exposed via the JS module and the xworker property is really an escape hatch for hacking around, not a field to deal with directly and at runtime on a live HTML page ... so kudos to Martin for finding eithr ways a solution that wasn't supposed to be offered (or working) but it does (due indirections, not intentionally as it was meant to be used) but definitively we want people to understand that if you want control over a worker you should initiate it within your Python code and from there offer, or call, any sync utility to the worker and from that worker.

it is feasable, my point is that it's also very footgun prone done that way as you don't have any guarantee that worker is handled just by one script and not more + that worker would bootstrap regardless of your mpy driver ... this is by design of how any worker works, and worked, to date, so what is wrong with the Python approach, where you write less, you have more control over pretty much everything, and no unexpected issues can happen? 🤔

I had a closer look ... basically you are moving the Python code bootstrap into HTML then you want workers to automatically give you those by id ... my point is that when you point at any ID that's not necessarily already bootstrapped from a MicroPython code point of view. It's true that MutationObserver is fast but if you expect that ID to always find a node down the HTML you might be at some point disappointed with bigger projects. This creates a precedent of non-scalable pattern for bigger projects where a MicroPython thing is bootstrapped ASAP and the pyodide worker is at the end of the page and not parsed yet by the browser. Nothing like this can ever happen if you bootstrap the worker within your code, as you own that worker at that point, it's not a "fantasy DOM element that might or might not be on the page at certain point in time" ... I don't know if this is clear or well explained, but one pattern, the PyWorker, always works, the other one has tons of caveats behind (ID element override by accident or duplicated, element not bootstrapped by pyscript core already, and so on).

Do we really want to pursue a potentially more problematic approach there? I also find the non relationship between a worker and its consumer hard to reason about ... a worker can live by its own, it doesn't need to be a zombie, it could have an advanced state (internally, in terms of data or operations) and your micropython driver (or js, or anything else) could arrive too late. None of this happens if your code bootstrap the worker.

Moreover ... we have worker terminal there and by no mean your other script should decide what happens in there as it cna be awaiting inputs or be fully unresponsive, all stuff that again won't happen if you bootstrap a worker for your utilities / usage.

If this is about shared workers then we should come up with a better solution, imho, otherwise everything looks ambiguous and error prone like this ... we globally leak any worker out there just because it has an ID attached? I'd say nope, that's not really cool.

to sum up my thoughts: this proposal is not bad per-se, but it's extremely ambiguous in terms of intent. If we want workers to be reachable through our namespace magic, we should think about a way that disambiguate the intent. Just adding an ID to a script element is not, in my opinion, a way to disambiguate such intent, and it will break if a worker terminal has an ID and that's used for other purposes ... we gotta find a better way, a better attribute, something not ambiguous, to enable this scenario to me. I get its simplicity or usefulness or expectation, I don't like its ambiguous nature and increased complexity over something that was initially planned to be the default for PyScript: everything should run out of a worker. We're not there yet because mpy bootstraps faster, but using workers is still a must do for any pyodide logic, not just as utility. I hope my thoughts are clear ... can we think of any better way to provide this functionality? One that's not ambiguous or in conflict with terminal too just because an ID is there? We could throw errors if that's the case, sure, but you see it's screaming for disambiguation when the scenario is "zombie worker" to server other scripts around (this, assuming I understood your use case ... if you want to interfere with also a terminal, that's a whole new world of "what ifs" to solve, imho).

counter-proposal: we add a worker-target="id" to the consumer of that worker and we provide there the workers["id"] out of our namespace ... that makes it clear from the consumer of the worker who wants to use what, we can await in a way or another that such target exists and it's bootstrapped and we can pass along the worker once that's ready ...

from pyscript import workers
worker = await workers["target-id"]

This forces users to think about who is consuming the worker, it forces them to add an id to the worker node, it makes the orchestration less magic and more explicit from our code perspective ... would this be good enough to you?

P.S. this would be a one-to-many relation for the worker element, but each consumer can handle only one worker per time ... we could have a worker-target="id1,id2" logic in place too though, if many-to-many relationship is desired.

Nope, the first one gets what the python worker code exports directly, the latter one returns a worker where you can expose sync methods to the code, reach its sync after awaiting the ready state, terminate the worker when desired / needed. These are already used production requirements so we better think about the former suggestion instead if we want 1:1 API, we can’t remove what’s out there and proven to be useful already.

Having re-read the discussion (with a coffee!) it seems that the difference between the <script> worker and PyWorker is in what gets exposed to the main thread? i.e. in the <script> element any module-scope attribute is available, whereas in PyWorker only those explicitly added onto the sync are available?

I’ll answer properly on Monday as it’s clear "we" lack understanding of the current offer and how people are already using PyWorker.

we shouldn’t lose features that have been asked already and are powerful:

• provide utilities to the worker from its owner
• being able to terminate a worker and bootstrap a new one if needed (memory constraints)

instead of breaking current features which cover 100% of use cases, we should think how to offer a simplification for the 80% of simple use cases ... heck I start thinking we shouldn’t even use the worker attribute on the HTML and think about a shared attribute instead.

every main can also ask same worker things but workers can be fully paused while serving other consumers so there are technical limitations to consider too. As example, a worker in a while loop that doesn’t block main can’t possibly ever serve another main thread script so we need to think about these details too ... I will provide concrete examples on Monday.