homepage

Since asm.js came on the scene, and now Web Assembly people have created CPython patches to support building CPython with emscripten. See:
* https://github.com/PeachPy/EmCPython -- Python 2.7
* https://github.com/dgym/cpython-emscripten/tree/master/3.5.2/patches -- Python 3.5.2
* https://github.com/iodide-project/pyodide/tree/master/cpython/patches -- Python 3.7.4
To ease the compiling of CPython with emscripten it would be helpful if patches that achieved these ends for the compiling to Web Assembly with emscripten were built into the upstream source repository itself.
If web assembly were to became a supported compilation target of the upstream CPython repository this would significantly reduce the friction of allowing CPython, and the latest CPython, to become a language readily usable within the browser.
Cheers,
Simon

Do you want to provide a pull request?

you can add 
* https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset -- Python 3.8.x 
(wasm not asm.js, clang-10+ required)
demo https://pmp-p.github.io/python-next/test.html
CPython can already run in the browser with very little patching, but major issues are :
 
 - asyncify'ing the whole wasm VM to have pre-emption over cPython's one to prevent blocking I/O slows down things *a lot* (10x)
=> (very?) bad user experience.
 - the size of vm + stdlib ~ 30 MiB and wasm compilation time.
=> bad user experience on first load or slow connexion.
 - the lack of threading in wasm MinimumViableProduct specification (but this is the browser standard for now), that leads to rewrite bits of stdlib ( like eg asyncio module )
=> adding more maintenance burden on stdlib (!)
i tested them all and my personnal opinion is : I can see no use case that would favour "stock" cPython wasm versus a blazing fast MicroPytho (or pycopy) wasm flavour or supercharged full stack pyodide.

Hi pmp-p and Serhiy,
I'd be more than happy to attempt a pull request, but I imagine a change
such as this needs to be discussed first, trying not to "rush to make a
patch" (https://www.youtube.com/watch?v=voXVTjwnn-U&feature=youtu.be&t=2546).
Also, I doubt I will do a good job of it... but I am more than happy to try.
A note regarding "supercharged full stack pyodide", potentially without
efforts such as upstreaming into CPython and emscripten the relevant
patches, that supercharged full stack may just unfortunately stagnate. See
https://github.com/iodide-project/pyodide/issues/635#issuecomment-613408912
With respect to blocking when running Python as WASM, I have found running
the WebAssembly CPython within a webworker and signalling data back and
forth causes there to be no UI issues. It ends up being quite a neat set
up. Main down side right now however is the set up is currently going
stale, hence me believing reaching out like this is in the best interests
of Python going forward.
Cheers,
Simon
On 2020年4月14日 at 18:58, pmp-p <report@bugs.python.org> wrote:
>
> pmp-p <pmpp.pub@gmail.com> added the comment:
>
> you can add
> *
> https://github.com/pmp-p/pydk/tree/master/sources.em/Python-3.8.0b4.patchset
> -- Python 3.8.x
>
> (wasm not asm.js, clang-10+ required)
>
> demo https://pmp-p.github.io/python-next/test.html
>
> CPython can already run in the browser with very little patching, but
> major issues are :
>
> - asyncify'ing the whole wasm VM to have pre-emption over cPython's one
> to prevent blocking I/O slows down things *a lot* (10x)
> => (very?) bad user experience.
>
> - the size of vm + stdlib ~ 30 MiB and wasm compilation time.
> => bad user experience on first load or slow connexion.
>
> - the lack of threading in wasm MinimumViableProduct specification (but
> this is the browser standard for now), that leads to rewrite bits of stdlib
> ( like eg asyncio module )
> => adding more maintenance burden on stdlib (!)
>
>
> i tested them all and my personnal opinion is : I can see no use case that
> would favour "stock" cPython wasm versus a blazing fast MicroPytho (or
> pycopy) wasm flavour or supercharged full stack pyodide.
>
> ----------
> nosy: +pmpp
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <https://bugs.python.org/issue40280>
> _______________________________________
>

I've been maintaining a Python Emscripten build for the Ren'Py (game engine) web port:
https://github.com/python-emscripten/python
https://renpy.beuc.net/
I recently tackled Python3 with a minimal/embeddable approach and checking the other ports already pointed in the discussion:
https://github.com/python-emscripten/python/tree/trunk/3.8/
(2 patches, and a short pyconfig.h fix-up)
There is also a Cython module to use the Emscripten C API.
Here's a demo at:
https://www.beuc.net/python-emscripten/demo/3/
(hello-world size: 3MB, with a few common modules: 4MB)
I can provide a pull request with a first few core changes.
Should this be done on github?
-
cross-compilation handling appears to follow an incorrect logic, in particular by querying 'dpkg' or parsing compiler output to detect include paths -- it is the (cross-)compiler's responsibility to provide the system paths, and detecting them manually causes conflicts. I had to patch setup.py. Let me know if I missed something.
Usually cross-compiling is triggered by non-matching build-type/host-type. Here cross-compilation logic is apparently triggered when exporting _PYTHON_HOST_PLATFORM=xxx manually (it's in the Makefile but not exported, and breaks normal build when exported). Is this the way it's meant to be used?

I think the first thing we should do is figure out whether we want to support Emscripten or WASI (or both).
Emscripten uses Javascript polyfills for some syscalls, while WASI makes direct calls the VM it is running in. They both can use WebAssembly for executing the code. This means Emscripten has wider API support, but WASI is lighter weight in many ways.
I think starting with patches to support Emscripten would be best, as it is easier to target, then add support for WASI later.
I think supporting WASI has a lot of value, because it can be run deterministically, which would be great for data science (Imagine a jupyter notebook that runs the same everywhere!)
One issue with WASI, and may be an issue with Emscripten, is threads. In 3.8 (or 3.9?) threadless builds were removed. However, WebAssembly's threading API is not really meant to emulate pthread, and SharedArrayBuffer, the primitive it is built on, is disabled in several browsers due to Spectre concerns.
Would patches to re-add a threadless build mode be accepted?

I have added wasm32/wasm64 architectures with emscripten/wasi operating system as cross-build targets. The values are based on Rust targets: 
$ rustc --print target-list | grep wasm
wasm32-unknown-emscripten
wasm32-unknown-unknown
wasm32-wasi
wasm64-unknown-unknown
wasm (WebAssembly) is "native instruction set" for the JavaScript VM while wasi or emscripten provide operating system facilities like memory management and I/O.

Our config.sub is recent enough and has support for wasm32, wasm64, wasi, and emscripten:
$ grep was[mi] config.sub 
 | wasm32 | wasm64 \
 | midnightbsd* | amdhsa* | unleashed* | emscripten* | wasi* \

LLVM considers `was32-wasi` an alias for `wasm32-unknown-wasi`. Verified on the WebAssembly Discover server at https://discord.com/channels/453584038356058112/596492540388179976/898618010221310062.

My last message had a couple of typos; should have been `wasm32-wasi` and "Discord", not "Discovery".

Do we need to care about our `config.guess` being updated as well? This is a totally ignorant question based on https://github.com/WebAssembly/wasi-sdk#notes-for-autoconf mentioning `config.guess`.

To help keep links up-to-date, Pyodide now lives at:
https://github.com/pyodide/pyodide/tree/main/cpython 

Our config.sub agrees with LLVM:
$ ./config.sub wasm32-wasi
wasm32-unknown-wasi
The config.sub and config.guess scripts in main are recent enough for wasm. Just to be sure I created https://github.com/python/cpython/pull/29781 and plan to backport the changeset to 3.10 and 3.9. It's generally safe to update the files to latest version.

New changeset 1052a39b7603e4d8401a5987af0c36f4a1d0b1e4 by Christian Heimes in branch 'main':
bpo-40280: Add wasm cross build targets (GH-29771)
https://github.com/python/cpython/commit/1052a39b7603e4d8401a5987af0c36f4a1d0b1e4

I have uploaded my config.site override to https://gist.github.com/tiran/5ccffa28723d3e4739db848451bd9efa . It contains overrides based on pyodide and overrides for new features.
I'm also getting this error with emscripten 2.0.13. _sys_shutdown is the syscall for shutdown(2) used by the socket module.
error: undefined symbol: __sys_shutdown (referenced by top-level compiled C/C++ code)
warning: Link with `-s LLD_REPORT_UNDEFINED` to get more information on undefined symbols
warning: To disable errors for undefined symbols use `-s ERROR_ON_UNDEFINED_SYMBOLS=0`
warning: ___sys_shutdown may need to be added to EXPORTED_FUNCTIONS if it arrives from a system library
Error: Aborting compilation due to previous errors

https://github.com/emscripten-core/emscripten/issues/13393 

New changeset ee1e2c604c8a66a407116d9c3e589ab0b9580c54 by Christian Heimes in branch 'main':
bpo-40280: Use Setup.stdlib static for wasm builds (GH-29784)
https://github.com/python/cpython/commit/ee1e2c604c8a66a407116d9c3e589ab0b9580c54

Thanks a lot for working on this!
> _sys_shutdown is the syscall for shutdown(2) used by the socket module.
Yes, the issue with Emscripten is that a number of system calls are either not implemented or implemented but not tested. See a list we are using in https://github.com/pyodide/pyodide/blob/main/cpython/pyconfig.undefs.h (though things might have improved since it was created).
FYI, with Emscripten, the list of CPython unit tests that are currently skipped (as of Python 3.9.5) is in https://github.com/pyodide/pyodide/blob/main/src/tests/python_tests.txt some of those are due to browser VM limitations (e.g. virtual filestem by Emscripten that's not fully POSIX compliant, no processes, no sockets, async only via the browser event loop etc), others because we are not yet using threading since not all browsers support it, and some failures probably need more investigation. 
Also opened https://github.com/pyodide/pyodide/issues/2000 . Let us know if there is anything we can do help with this effort.

New changeset d224e769b83a6a027bec8e21ee62db2b96d5af8e by Christian Heimes in branch 'main':
bpo-40280: clean and ignore .wasm files (GH-29794)
https://github.com/python/cpython/commit/d224e769b83a6a027bec8e21ee62db2b96d5af8e

Thanks Roman, I replied on the pyodide issue tracker.

New changeset 8caceb7a474bf32cddfd25fba25b531ff65f4365 by Christian Heimes in branch 'main':
bpo-40280: Add configure check for socket shutdown (GH-29795)
https://github.com/python/cpython/commit/8caceb7a474bf32cddfd25fba25b531ff65f4365

New changeset 4ebde73b8e416eeb1fd5d2ca3283f7ddb534c5b1 by Christian Heimes in branch 'main':
bpo-40280: Move hard-coded feature checks to configure (GH-29789)
https://github.com/python/cpython/commit/4ebde73b8e416eeb1fd5d2ca3283f7ddb534c5b1

New changeset b3f443a35e08a473d52868621e9b268259139cb2 by Erlend Egeberg Aasland in branch 'main':
bpo-40280: Disable unusable core extension modules on emscripten (GH-29834)
https://github.com/python/cpython/commit/b3f443a35e08a473d52868621e9b268259139cb2

New changeset 6ac3c8a3140c17bd71ba98dfc5250c371101e77c by Christian Heimes in branch 'main':
bpo-40280: Emscripten systems use .wasm suffix by default (GH-29842)
https://github.com/python/cpython/commit/6ac3c8a3140c17bd71ba98dfc5250c371101e77c

New changeset 9deb83468c56c7484645e6e3a6d0183cd6a0afd7 by Christian Heimes in branch 'main':
bpo-40280: Emscripten defaults to --with-ensurepip=no (GH-29873)
https://github.com/python/cpython/commit/9deb83468c56c7484645e6e3a6d0183cd6a0afd7

New changeset 309110f37cdfc78d160ed08ae8faa6f6160ba87e by Christian Heimes in branch 'main':
bpo-40280: Emscripten with_ensurepip=no, second attempt (GH-29884)
https://github.com/python/cpython/commit/309110f37cdfc78d160ed08ae8faa6f6160ba87e

New changeset cb2b3c8d3566ae46b3b8d0718019e1c98484589e by Christian Heimes in branch 'main':
bpo-40280: Emscripten has no support for subprocesses (GH-29872)
https://github.com/python/cpython/commit/cb2b3c8d3566ae46b3b8d0718019e1c98484589e

New changeset cb8f491f46e262549f6c447b31625cab7c20a60a by Christian Heimes in branch 'main':
bpo-40280: Optimize ints and and startup on wasm (GH-29887)
https://github.com/python/cpython/commit/cb8f491f46e262549f6c447b31625cab7c20a60a

New changeset a31173c5ceb1708df687f942d714bdecae7cb759 by Christian Heimes in branch 'main':
bpo-40280: Update what's new (GH-29893)
https://github.com/python/cpython/commit/a31173c5ceb1708df687f942d714bdecae7cb759

New changeset 0339434835aa74dc78a38ae12ea7d2973c144eb1 by Christian Heimes in branch 'main':
bpo-40280: Add Tools/wasm with helpers for cross building (GH-29984)
https://github.com/python/cpython/commit/0339434835aa74dc78a38ae12ea7d2973c144eb1

New changeset 5c66414b5561c54e7a0f4bde8cc3271908ea525e by Ethan Smith in branch 'main':
bpo-40280: Disable epoll_create in Emscripten config.site (GH-30494)
https://github.com/python/cpython/commit/5c66414b5561c54e7a0f4bde8cc3271908ea525e

New changeset 43839ba438368a50f22f718d4ce8ce607c17046c by Christian Heimes in branch 'main':
bpo-40280: Add --with-emscripten-target to build for browser or node (GH-30552)
https://github.com/python/cpython/commit/43839ba438368a50f22f718d4ce8ce607c17046c

New changeset e34c9367f8e0068ca4bcad9fb5c2c1024d02a77d by Christian Heimes in branch 'main':
bpo-40280: Allow to compile _testcapi as builtin module (GH-30559)
https://github.com/python/cpython/commit/e34c9367f8e0068ca4bcad9fb5c2c1024d02a77d

New changeset c8319f7921fbcb0dea04da48a1b04a5d0d21ae1c by Christian Heimes in branch 'main':
bpo-40280: Build WASM stdlib bundle and more modules for node (GH-30597)
https://github.com/python/cpython/commit/c8319f7921fbcb0dea04da48a1b04a5d0d21ae1c

New changeset ee1a8b336d30476e9635a6826f61a99fc3604159 by Christian Heimes in branch 'main':
bpo-40280: Block more syscalls that are causing crashes in tests (GH-30601)
https://github.com/python/cpython/commit/ee1a8b336d30476e9635a6826f61a99fc3604159

A bunch of failing tests are related to bpo-46390. CPython's test suite does not pass with musl libc yet.

New changeset 91e33ac3d08a1c6004c469da2c0e2a97b5bdc53c by Christian Heimes in branch 'main':
bpo-40280: Add requires_fork test helper (GH-30622)
https://github.com/python/cpython/commit/91e33ac3d08a1c6004c469da2c0e2a97b5bdc53c

New changeset 7f4b69b9076bdbcea31f6ad16eb125ee99cf0175 by Christian Heimes in branch 'main':
bpo-40280: Change subprocess imports for cleaner error on wasm32 (GH-30620)
https://github.com/python/cpython/commit/7f4b69b9076bdbcea31f6ad16eb125ee99cf0175

New changeset c02e860ee79f29905be6fca997c96bb1a404bb32 by Christian Heimes in branch 'main':
bpo-40280: Misc fixes for wasm32-emscripten (GH-30722)
https://github.com/python/cpython/commit/c02e860ee79f29905be6fca997c96bb1a404bb32

New changeset d5fd438b38248a0d2e91898475369361e34f74b7 by Christian Heimes in branch 'main':
bpo-40280: Get help() working and more (GH-30858)
https://github.com/python/cpython/commit/d5fd438b38248a0d2e91898475369361e34f74b7

New changeset 8464fbc42ecc9ce504faac499711dcdc6eedef16 by Christian Heimes in branch 'main':
bpo-40280: Skip subprocess-based tests on wasm32-emscripten (GH-30615)
https://github.com/python/cpython/commit/8464fbc42ecc9ce504faac499711dcdc6eedef16

New changeset 606e496dd6e2ace298532da200169124c26ae0f2 by Christian Heimes in branch 'main':
bpo-40280: Use presence of msvcrt module to detect Windows (GH-30930)
https://github.com/python/cpython/commit/606e496dd6e2ace298532da200169124c26ae0f2

New changeset 96b344c2f15cb09251018f57f19643fe20637392 by Christian Heimes in branch 'main':
bpo-40280: Address more test failures on Emscripten (GH-31050)
https://github.com/python/cpython/commit/96b344c2f15cb09251018f57f19643fe20637392

New changeset 5081e78efde901556398615eb477c63c836686e5 by Christian Heimes in branch 'main':
bpo-40280: Block more non-working syscalls in Emscripten (GH-31757)
https://github.com/python/cpython/commit/5081e78efde901556398615eb477c63c836686e5

I have fixed a signal-related issue in GH-31768 / bpo-23325 

New changeset 95ba723c54ab52487bc9f986b651536cfae15b29 by Christian Heimes in branch 'main':
bpo-40280: Skip dysfunctional pipe tests on Emscripten (GH-31770)
https://github.com/python/cpython/commit/95ba723c54ab52487bc9f986b651536cfae15b29

New changeset de554d6e02228b840eb6bffaf7d406c0ef368d5f by Christian Heimes in branch 'main':
bpo-40280: Skip more tests/features that don't apply to Emscripten (GH-31791)
https://github.com/python/cpython/commit/de554d6e02228b840eb6bffaf7d406c0ef368d5f

New changeset ecfff63e06e77e22035a7f7caa26986f033f3aea by Christian Heimes in branch 'main':
bpo-40280: Disable AF_UNIX, AF_PACKET, SO_REUSE* on Emscripten (#31829)
https://github.com/python/cpython/commit/ecfff63e06e77e22035a7f7caa26986f033f3aea

New changeset f00ced8396f2d7683e58b9d5ebbf5797992bf477 by Christian Heimes in branch 'main':
bpo-40280: select: Use NULL for empty fdset (GH-31865)
https://github.com/python/cpython/commit/f00ced8396f2d7683e58b9d5ebbf5797992bf477

New changeset b43b9b49be7d42d2826106c719d1e51f0776be0a by Christian Heimes in branch 'main':
bpo-40280: Skip wakeup_fd pipe tests on Emscripten (GH-31909)
https://github.com/python/cpython/commit/b43b9b49be7d42d2826106c719d1e51f0776be0a

New changeset a4674f0194067a801f6c6bdb4fc6448e3a40e069 by Christian Heimes in branch 'main':
bpo-40280: Detect presence of time.tzset and thread_time clock (GH-31898)
https://github.com/python/cpython/commit/a4674f0194067a801f6c6bdb4fc6448e3a40e069

New changeset ef1327e3b622e0cafdf8bfc1f480fed0dd386be6 by Christian Heimes in branch 'main':
bpo-40280: Skip more tests on Emscripten (GH-31947)
https://github.com/python/cpython/commit/ef1327e3b622e0cafdf8bfc1f480fed0dd386be6

New changeset deeaac49e267285158264643799624623f4a7b29 by Christian Heimes in branch 'main':
bpo-40280: Skip socket, fork, subprocess tests on Emscripten (GH-31986)
https://github.com/python/cpython/commit/deeaac49e267285158264643799624623f4a7b29

New changeset 8a0a9e5b1928fab7d9819c8d6498ef5c0b9383af by Christian Heimes in branch 'main':
bpo-40280: Add wasm32-emscripten and wasm32-wasi SOABI (GH-32095)
https://github.com/python/cpython/commit/8a0a9e5b1928fab7d9819c8d6498ef5c0b9383af

New changeset 17245c815e44e79d4dad6a99b54000956a4a7229 by Christian Heimes in branch 'main':
bpo-40280: Add debug Emscripten flavors (GH-32233)
https://github.com/python/cpython/commit/17245c815e44e79d4dad6a99b54000956a4a7229

New changeset 082d3495d0c820972f09f6109a98ed7eb5a7b79f by Christian Heimes in branch 'main':
bpo-40280: Emscripten fork_exec now fails early (GH-32224)
https://github.com/python/cpython/commit/082d3495d0c820972f09f6109a98ed7eb5a7b79f

New changeset 59be9cd748728b03ac61287681c3010bcec5e558 by Christian Heimes in branch 'main':
bpo-40280: Detect if WASM platform supports threading (GH-32243)
https://github.com/python/cpython/commit/59be9cd748728b03ac61287681c3010bcec5e558

New changeset 4ed8a9a589d2eee7442e0c9417515a707e504faa by Christian Heimes in branch 'main':
bpo-40280: Enable most file-at() and nanosleep APIs again (GH-32238)
https://github.com/python/cpython/commit/4ed8a9a589d2eee7442e0c9417515a707e504faa

pmp-p wrote on GH-32253:
when using --enable-wasm-dynamic-linking i think --enable-shared should set -fPIC and -s SIDE_MODULE=1 for libpython.so
I have not tried --enable-shared with Emscripten yet. So far I have assumed that the result is a fat Python binary that can load additional shared libraries.

New changeset c9844cb8aa9615cdc8770d1e43ce6e2ac3efd836 by Christian Heimes in branch 'main':
bpo-40280: Add --enable-wasm-dynamic-linking (GH-32253)
https://github.com/python/cpython/commit/c9844cb8aa9615cdc8770d1e43ce6e2ac3efd836

New changeset 96e09837fb8031aebe8d823dd19ef664a34bcfad by Christian Heimes in branch 'main':
bpo-40280: Add limited Emscripten REPL (GH-32284)
https://github.com/python/cpython/commit/96e09837fb8031aebe8d823dd19ef664a34bcfad

New changeset 765f6dee0fcf55c6ea258c2be4cc9dfb1b014f60 by Christian Heimes in branch 'main':
bpo-40280: WASM defaults to no dynamic linking (GH-32360)
https://github.com/python/cpython/commit/765f6dee0fcf55c6ea258c2be4cc9dfb1b014f60

New changeset 2b16a08bc77475917dd5c96417aef4c5210b45ac by Christian Heimes in branch 'main':
bpo-40280: Detect missing threading on WASM platforms (GH-32352)
https://github.com/python/cpython/commit/2b16a08bc77475917dd5c96417aef4c5210b45ac

Christian: IDLE and test_idle require threading. I once tested for threading in test_idle (not sure about IDLE itself) but removed the test when threading was made supposedly non-optional. Test_idle still tests for tkinter and idlelib. For PR 32352. did you miss IDLE's requirement or is tkinter and hence IDLE and its test not included on WASM/emscripten? Will it never be?

In GH-32352 I changed only tests that I was able to execute under wasm32-emscripten. As far as I know TK does not build for WASM. Somebody would have to port TCL/TK to WebGL or SDL2 to make it work in the browser.