This issue tracker has been migrated to GitHub ,
and is currently read-only.
For more information,
see the GitHub FAQs in the Python's Developer Guide.
Created on 2021年04月26日 20:59 by Genarito, last changed 2022年04月11日 14:59 by admin.
| Messages (8) | |||
|---|---|---|---|
| msg391985 - (view) | Author: Genaro Camele (Genarito) | Date: 2021年04月26日 20:59 | |
I've a piece of code that submits a task to a [ThreadPoolExecutor][1] which starts a [Process][2]. I've realised that in Python 3.8 that Process finished with exit code `0`. But I've updated Python to the 3.9 version and this started to finishing with exit code `1`! Even when the Process executes an empty task. Here's a minimal example: ```python from multiprocessing import Process from concurrent.futures import ThreadPoolExecutor def some_task(): pass def execute_error(): p = Process(target=some_task) p.start() p.join() print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!! executor = ThreadPoolExecutor(max_workers=4) executor.submit(execute_error) # execute_error() # IMPORTANT: this works correctly (exit 0) ``` My versions: ``` Ubuntu 21.04 Python 3.9.4 ``` **Note** that if `__execute_error` is called outside the ThreadPoolExecutor it works correctly. Running on Python 3.8.6 exitcode = 0 too. [1]: https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor [2]: https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.Process |
|||
| msg396522 - (view) | Author: Alexandre Sicard (sicard_elda) | Date: 2021年06月25日 10:16 | |
Thank you very much for this report, Genaro. I encountered the same bug with a Process running in the context of a Django view. Downgrading to Python 3.8 also fixed the issue for me. Versions: python:3.9-alpine Docker image, running Python 3.9.5 and Alpine 3.13. Can also reproduce on the standard (Debian Buster based) python:3.9 image, and on Arch Linux (bare metal), all also running Python 3.9.5. |
|||
| msg398157 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021年07月24日 14:11 | |
I am working on a fix for this bug. I'm a beginner cpython contributor, so if anyone recognizes this as a fool's errand, please let me know! |
|||
| msg398159 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021年07月24日 14:30 | |
Ah never mind. @Genarito, the ThreadPoolExecutor is supposed to be used as a context manager. In your current code, the script ends and Python starts tearing itself down while `execute_error` is still running in a subprocess. If you simply use the ThreadPoolExecutor to a context manager, the error goes away:: ```python from multiprocessing import Process from concurrent.futures import ThreadPoolExecutor def some_task(): pass def execute_error(): p = Process(target=some_task) p.start() p.join() print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!! # THIS IS THE IMPORTANT CHANGE with ThreadPoolExecutor(max_workers=4) as executor: executor.submit(execute_error) ``` |
|||
| msg398175 - (view) | Author: Genaro Camele (Genarito) | Date: 2021年07月24日 21:55 | |
Hi @jack__d, thanks for your answer and time. Unfortunately, It's still a regression, as in Python < 3.9 my example works as expected |
|||
| msg398202 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021年07月26日 00:24 | |
I've identified the first bad commit with git-bisect: commit b61b818d916942aad1f8f3e33181801c4a1ed14b Author: Kyle Stanley <aeros167@gmail.com> Date: Fri Mar 27 15:31:22 2020 -0400 bpo-39812: Remove daemon threads in concurrent.futures (GH-19149) Remove daemon threads from :mod:`concurrent.futures` by adding an internal `threading._register_atexit()`, which calls registered functions prior to joining all non-daemon threads. This allows for compatibility with subinterpreters, which don't support daemon threads. |
|||
| msg398203 - (view) | Author: Jack DeVries (jack__d) * | Date: 2021年07月26日 00:25 | |
The first bad commit was a fix for bpo-39812. |
|||
| msg416966 - (view) | Author: Thomas Grainger (graingert) * | Date: 2022年04月08日 09:35 | |
the problem is multiprocessing/process is calling threading._shutdown which tries to join its own thread, because concurrent.futures.thread._threads_queues contains the main thread in the subprocess File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/multiprocessing/process.py", line 333, in _bootstrap threading._shutdown() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1530, in _shutdown atexit_call() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit t.join() File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1086, in join raise RuntimeError("cannot join current thread") |
|||
| History | |||
|---|---|---|---|
| Date | User | Action | Args |
| 2022年04月11日 14:59:44 | admin | set | github: 88110 |
| 2022年04月08日 09:35:24 | graingert | set | nosy:
+ graingert messages: + msg416966 |
| 2021年07月26日 00:25:22 | jack__d | set | messages: + msg398203 |
| 2021年07月26日 00:24:24 | jack__d | set | messages: + msg398202 |
| 2021年07月24日 21:55:55 | Genarito | set | messages: + msg398175 |
| 2021年07月24日 14:30:25 | jack__d | set | messages: + msg398159 |
| 2021年07月24日 14:11:03 | jack__d | set | nosy:
+ jack__d messages: + msg398157 |
| 2021年06月25日 10:16:24 | sicard_elda | set | nosy:
+ sicard_elda messages: + msg396522 |
| 2021年04月30日 22:15:35 | terry.reedy | set | nosy:
+ pitrou, davin |
| 2021年04月26日 20:59:45 | Genarito | create | |