homepage

> And setup.py does use use multiple processes via a thread pool, where each thread calls fork(), if it detects that "make" was invoked with the parallel (-j) option.
Oh ok, now I get it :-) setup.py uses distutils to build extensions and distutils.command.build_ext uses concurrent.futures for parallelism:
 if self.parallel:
 self._build_extensions_parallel()
 else:
 self._build_extensions_serial()
with:
 def _build_extensions_parallel(self):
 workers = self.parallel
 if self.parallel is True:
 workers = os.cpu_count() # may return None
 try:
 from concurrent.futures import ThreadPoolExecutor
 except ImportError:
 workers = None
 if workers is None:
 self._build_extensions_serial()
 return
 with ThreadPoolExecutor(max_workers=workers) as executor:
 futures = [executor.submit(self.build_extension, ext)
 for ext in self.extensions]
 for ext, fut in zip(self.extensions, futures):
 with self._filter_build_errors(ext):
 fut.result()
The problem is not concurrent.futures but the job submitted to the executor: build_extension() uses distutils.spawn() which is unsafe.
distutils.spawn() is the root issue if I understood correctly. It must be rewritten with subprocess.