homepage

This issue tracker has been migrated to GitHub , and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
Title: Processes in Python 3.9 exiting with code 1 when It's created inside a ThreadPoolExecutor
Type: behavior Stage:
Components: Versions: Python 3.9
process
Status: open Resolution:
Dependencies: Superseder:
Assigned To: Nosy List: Genarito, davin, graingert, jack__d, pitrou, sicard_elda
Priority: normal Keywords:

Created on 2021年04月26日 20:59 by Genarito, last changed 2022年04月11日 14:59 by admin.

Messages (8)
msg391985 - (view) Author: Genaro Camele (Genarito) Date: 2021年04月26日 20:59
I've a piece of code that submits a task to a [ThreadPoolExecutor][1] which starts a [Process][2]. I've realised that in Python 3.8 that Process finished with exit code `0`. But I've updated Python to the 3.9 version and this started to finishing with exit code `1`! Even when the Process executes an empty task.
Here's a minimal example:
```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor
def some_task():
 pass
def execute_error():
 p = Process(target=some_task)
 p.start()
 p.join()
 print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!!
executor = ThreadPoolExecutor(max_workers=4)
executor.submit(execute_error)
# execute_error() # IMPORTANT: this works correctly (exit 0)
```
My versions:
```
Ubuntu 21.04
Python 3.9.4
```
**Note** that if `__execute_error` is called outside the ThreadPoolExecutor it works correctly.
Running on Python 3.8.6 exitcode = 0 too.
 [1]: https://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor
 [2]: https://docs.python.org/3.9/library/multiprocessing.html#multiprocessing.Process 
msg396522 - (view) Author: Alexandre Sicard (sicard_elda) Date: 2021年06月25日 10:16
Thank you very much for this report, Genaro. I encountered the same bug with a Process running in the context of a Django view. Downgrading to Python 3.8 also fixed the issue for me.
Versions:
python:3.9-alpine Docker image, running Python 3.9.5 and Alpine 3.13. Can also reproduce on the standard (Debian Buster based) python:3.9 image, and on Arch Linux (bare metal), all also running Python 3.9.5.
msg398157 - (view) Author: Jack DeVries (jack__d) * Date: 2021年07月24日 14:11
I am working on a fix for this bug. I'm a beginner cpython contributor, so if anyone recognizes this as a fool's errand, please let me know!
msg398159 - (view) Author: Jack DeVries (jack__d) * Date: 2021年07月24日 14:30
Ah never mind. @Genarito, the ThreadPoolExecutor is supposed to be used as a context manager. In your current code, the script ends and Python starts tearing itself down while `execute_error` is still running in a subprocess.
If you simply use the ThreadPoolExecutor to a context manager, the error goes away::
```python
from multiprocessing import Process
from concurrent.futures import ThreadPoolExecutor
def some_task():
 pass
def execute_error():
 p = Process(target=some_task)
 p.start()
 p.join()
 print(p.exitcode) # This is always 1 on a ThreadPoolExecutor!!!
# THIS IS THE IMPORTANT CHANGE
with ThreadPoolExecutor(max_workers=4) as executor:
 executor.submit(execute_error)
```
msg398175 - (view) Author: Genaro Camele (Genarito) Date: 2021年07月24日 21:55
Hi @jack__d, thanks for your answer and time. Unfortunately, It's still a regression, as in Python < 3.9 my example works as expected
msg398202 - (view) Author: Jack DeVries (jack__d) * Date: 2021年07月26日 00:24
I've identified the first bad commit with git-bisect:
commit b61b818d916942aad1f8f3e33181801c4a1ed14b
Author: Kyle Stanley <aeros167@gmail.com>
Date: Fri Mar 27 15:31:22 2020 -0400
 bpo-39812: Remove daemon threads in concurrent.futures (GH-19149)
 Remove daemon threads from :mod:`concurrent.futures` by adding
 an internal `threading._register_atexit()`, which calls registered functions
 prior to joining all non-daemon threads. This allows for compatibility
 with subinterpreters, which don't support daemon threads.
msg398203 - (view) Author: Jack DeVries (jack__d) * Date: 2021年07月26日 00:25
The first bad commit was a fix for bpo-39812.
msg416966 - (view) Author: Thomas Grainger (graingert) * Date: 2022年04月08日 09:35
the problem is multiprocessing/process is calling threading._shutdown which tries to join its own thread, because concurrent.futures.thread._threads_queues contains the main thread in the subprocess
 File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/multiprocessing/process.py", line 333, in _bootstrap
 threading._shutdown()
 File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1530, in _shutdown
 atexit_call()
 File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/concurrent/futures/thread.py", line 31, in _python_exit
 t.join()
 File "/home/graingert/miniconda3/envs/dask-distributed/lib/python3.10/threading.py", line 1086, in join
 raise RuntimeError("cannot join current thread")
History
Date User Action Args
2022年04月11日 14:59:44adminsetgithub: 88110
2022年04月08日 09:35:24graingertsetnosy: + graingert
messages: + msg416966
2021年07月26日 00:25:22jack__dsetmessages: + msg398203
2021年07月26日 00:24:24jack__dsetmessages: + msg398202
2021年07月24日 21:55:55Genaritosetmessages: + msg398175
2021年07月24日 14:30:25jack__dsetmessages: + msg398159
2021年07月24日 14:11:03jack__dsetnosy: + jack__d
messages: + msg398157
2021年06月25日 10:16:24sicard_eldasetnosy: + sicard_elda
messages: + msg396522
2021年04月30日 22:15:35terry.reedysetnosy: + pitrou, davin
2021年04月26日 20:59:45Genaritocreate

AltStyle によって変換されたページ (->オリジナル) /