homepage

Hello,
I found this bit in my inbox, I forgot why I cared about it, but it raises an exception (at least on Windows):
>>> import multiprocessing
>>> p = multiprocessing.Process(target=bytes.maketrans, args=(b'abc', b'xyz'))
>>> p.start()
Traceback (most recent call last):
 File "C:\Python32\Lib\pickle.py", line 679, in save_global
 klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'maketrans'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "C:\Python32\Lib\multiprocessing\process.py", line 130, in start
 self._popen = Popen(self)
 File "C:\Python32\Lib\multiprocessing\forking.py", line 267, in __init__
 dump(process_obj, to_child, HIGHEST_PROTOCOL)
 File "C:\Python32\Lib\multiprocessing\forking.py", line 190, in dump
 ForkingPickler(file, protocol).dump(obj)
 File "C:\Python32\Lib\pickle.py", line 237, in dump
 self.save(obj)
 File "C:\Python32\Lib\pickle.py", line 344, in save
 self.save_reduce(obj=obj, *rv)
 File "C:\Python32\Lib\pickle.py", line 432, in save_reduce
 save(state)
 File "C:\Python32\Lib\pickle.py", line 299, in save
 f(self, obj) # Call unbound method with explicit self
 File "C:\Python32\Lib\pickle.py", line 623, in save_dict
 self._batch_setitems(obj.items())
 File "C:\Python32\Lib\pickle.py", line 656, in _batch_setitems
 save(v)
 File "C:\Python32\Lib\pickle.py", line 299, in save
 f(self, obj) # Call unbound method with explicit self
 File "C:\Python32\Lib\pickle.py", line 683, in save_global
 (obj, module, name))
_pickle.PicklingError: Can't pickle <built-in function maketrans>: it's not found as __main__.maketrans
If you do the same things with `threading.Thread`, it works, but for `multiprocessing.Process` it doesn't. Is this a general problem with pickling "unbound methods"?

Running on winxp with IDLE, I get the second traceback, all the same after the first line. Given "target is the callable object to be invoked by the run() method.", I would have expected this to work too.
Problem is not builtins:
class C:
 def f(s): print( 'here')
p = multiprocessing.Process(target=C.f, args=(C(),))
p.start()
gives same error, ending in
_pickle.PicklingError: Can't pickle <function f at 0x00FE5AE0>: it's not found as __main__.f

Do things like this in the REPL are known not to work. Things are not protected in the if __name__ block so we can import the script properly to run it.

Test attached.

Why is this still marked as "test needed"?

The test should be a diff against Lib/test/test_multiprocessing.py that adds a unit test.

Diff attached, is it good? I'm not very experienced with diffs, I usually work with pull requests.

Your patch is good in this case, as the person who applies the patch knows which file is affected.
In the future, use diff -u original_file modified_file to get a unified diff. It's the "de facto" format for patches.

Actually, I only commented on the patch format and not on the actual contents of the patch, sorry :)
Your test method missed the self parameter, and the test case needed to be added to the testcases_other list for the test to be actually run. I attached a corrected patch.
And, now that I tested it, the test runs correctly on Linux, so maybe this is a Windows specific issue?

Thanks for the `-u` tip and the correction to the code, Petri. I removed my previous files since yours is the definite one. And yeah, it's a Windows issue.

Jesse, I do not understand your comment, including 'REPL'

REPL is the Python interactive prompt in this case (REPL is Read Eval Print Loop). So Jesse is saying that using multiprocessing from the REPL (at least on Windows) isn't supported. This is because on Windows multiprocessing needs to re-import the main program in order to start the function in a worker process. When using the REPL, there is no main program from which to import the function.
The test case failing on windows may or may not be a related issue; I'm not familiar enough with mulitprocessing to say.

After looking at the doc chapter, I get that 'if __name__' block is needed on Windows.
OK. batch mode with if __name__ block:
testmp.py
---------
print('Top of Module')
class C:
 def f(s): print('Method C.f')
if __name__ == '__main__':
 print('Start of main block')
 import multiprocessing
 p = multiprocessing.Process(target=C.f, args=(C(),))
 p.start()
 p.join()
command-prompt..\python32> python @misc2\testmp.py
output (similar to that of OP):
Top of Module
Start of main block
Traceback (most recent call last):
 File "C:\Programs\Python32\lib\pickle.py", line 679, in save_global
 klass = getattr(mod, name)
AttributeError: 'module' object has no attribute 'f'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
 File "@misc2\testmp.py", line 9, in <module>
 p.start()
 File "C:\Programs\Python32\lib\multiprocessing\process.py", line 130, in start
 self._popen = Popen(self)
 File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 267, in __init__
 dump(process_obj, to_child, HIGHEST_PROTOCOL)
 File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 190, in dump
 ForkingPickler(file, protocol).dump(obj)
 File "C:\Programs\Python32\lib\pickle.py", line 237, in dump
 self.save(obj)
 File "C:\Programs\Python32\lib\pickle.py", line 344, in save
 self.save_reduce(obj=obj, *rv)
 File "C:\Programs\Python32\lib\pickle.py", line 432, in save_reduce
 save(state)
 File "C:\Programs\Python32\lib\pickle.py", line 299, in save
 f(self, obj) # Call unbound method with explicit self
 File "C:\Programs\Python32\lib\pickle.py", line 623, in save_dict
 self._batch_setitems(obj.items())
 File "C:\Programs\Python32\lib\pickle.py", line 656, in batch_setitems
 save(v)
 File "C:\Programs\Python32\lib\pickle.py", line 299, in save
 f(self, obj) # Call unbound method with explicit self
 File "C:\Programs\Python32\lib\pickle.py", line 683, in save_global
 (obj, module, name))
_pickle.PicklingError: Can't pickle <function f at 0x00B4B4B0>: it's not found as __main__.f
C:\Programs\Python32>Top of Module
Traceback (most recent call last):
 File "<string>", line 1, in <module>
 File "C:\Programs\Python32\lib\multiprocessing\forking.py", line 370, in main
 self = load(from_parent)
EOFError
The essential problem seems to be the attempt to pickle the method attribute of the class as a class attribute of the module.

This works perfectly on 64 bit Windows 8.1 for 3.4.1 and 3.5.0a0.

Confirmed here it's working in Python 3.4, I guess it was fixed sometime in the last few years.
I guess the only thing we'd care about now is ensuring a test for this was added to the test suite, so there wouldn't be a regression. Can anyone confirm that?

Pickling of builtin functions and methods was indeed improved thanks to __qualname__ support. Closing.