homepage

This issue tracks the work pending for the inclusion of the pyprocessing 
module under the multiprocessing name for Python 2.6 and 3.0. If needed, 
additional tickets will be created for issues which will need to be 
addressed after initial inclusion.
Currently open:
- Redo all documentation to match ReST format of stdlib
- Update docs to reflect new PEP8 names
- Update all unit tests to be in Unittest format
Closed:
- Rename module API to be PEP8 compliant (roudkerk)
- Link csource libs into python-trunk build (jnoller)
- redo all local "." style imports to reflect the new namespace 
(jnoller)

Quick status:
Completed:
- Redo all documentation to match ReST format of stdlib
 - thanks to Benjamin Peterson for the massive conversion he did.
- All unit tests are converted, I'm just chasing down a few lingering 
bugs that came with the conversion. (jnoller)
Richard and I are ironing out a few remaining API cleanups that came 
about with the renaming and I'll update the docs to reflect the new 
method names.

I'm attaching my version of the docs. (Jesse, I did fix the Sphinx
warnings and some other problems I noticed.) Georg, if you could cast
your expert reST eye over them, I'd be much obliged.

Attaching reviewed docs patch.

I applied Ben's updated patch and then applied Georg's patch - but now I 
am getting an error doing "make html", unfortunately sphinx/ReST doesn't 
include line numbers with the error:
mify library/miniaeframe library/misc library/mm library/mmap 
library/modulefinder library/modules library/msilib library/msvcrt 
library/multifile library/multiprocessing reST markup error:
/Users/jesse/open_source/subversion/python-
trunk/Doc/library/multiprocessing.rst:: (SEVERE/4) Duplicate ID: 
"module-multiprocessing".
make: *** [build] Error 1
Ben/Georg - did I apply them wrong?

Do this:
svn revert -R .;
rm Doc/library/multiprocessing.rst
rm Doc/includes/mp*.py
Then apply just Georg's patch. That should work.

Thanks Ben - even with that, for some reason I ended up with duplicate 
chunks and data inside of the .rst file - wiping it all out, and then 
making a blank version of the .rst and then applying the patch from Georg 
worked.

Attached is rev 1 of a complete patch for inclusion in 2.6. This patch 
was developed off of trunk and includes the unit tests, docs, modules, c 
module, etc. 
2 of the unit tests had to be commented out, the first due to the 
difference in the threading versus process API (is_set vs. isSet) and 
the second is an OS/X issue I am working with Richard on.
Note, some of the examples may be out of date - I am working on those.
A full review of this would be appreciated. Pending the application of 
the patch in issue 1638, this initial revision could be applied to trunk 
for the wednesday build.

Added to the code review system too: http://codereview.appspot.com/2061 

How much work needs to be done to port this to Py3k?

Richard mentioned it should just work after being run though lib2to3

On Tue, Jun 10, 2008 at 3:50 PM, Jesse Noller <report@bugs.python.org> wrote:
>
> Jesse Noller <jnoller@gmail.com> added the comment:
>
> Richard mentioned it should just work after being run though lib2to3
Excellent! That clears up a main worry of mine.

Congrats, you're in! See r64104 in the trunk.

Please see Georg's Python-dev message about importing multiprocessing.

Specifically:
>>> import multiprocessing
Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Users/jesse/open_source/subversion/python-
trunk/Lib/multiprocessing/__init__.py", line 63, in <module>
import _multiprocessing
AttributeError: 'module' object has no attribute 'BufferTooShort'
This occurs due to the fact that __init__.py imports _multiprocessing 
prior to the definition of the exceptions. This should be moved to the 
ccode. From Georg:
"The test suite passes (at least for some buildbots) because it imports
_multiprocessing first, which then in its init function imports 
multiprocessing
to get the BufferTooShort exception.
Since BufferTooShort and other exceptions inheriting from ProcessError 
are
simple derived exceptions, why aren't they created in the C module in 
the
first place?"

Additional things to clean up post-beta:
- Review all docs fully to ensure the documented API is still in sync with 
the renames.
- Restore the commented out tests in test_event
- Restore the commented out test in test_bounded_semaphore

The _multiprocessing module is not building under Windows at the moment.
Attempting to import multiprocessing (from an .exe build from the
current svn) gives "ImportError: No module named _multiprocessing" and
the test suite skips the test for the same reason.

On Wed, Jun 11, 2008 at 9:20 AM, Tim Golden <report@bugs.python.org> wrote:
>
> Tim Golden <mail@timgolden.me.uk> added the comment:
>
> The _multiprocessing module is not building under Windows at the moment.
> Attempting to import multiprocessing (from an .exe build from the
> current svn) gives "ImportError: No module named _multiprocessing" and
> the test suite skips the test for the same reason.
Can you tell why it's not building?

Benjamin Peterson wrote:
> Benjamin Peterson <musiccomposition@gmail.com> added the comment:
> 
> On Wed, Jun 11, 2008 at 9:20 AM, Tim Golden <report@bugs.python.org> wrote:
>> Tim Golden <mail@timgolden.me.uk> added the comment:
>>
>> The _multiprocessing module is not building under Windows at the moment.
>> Attempting to import multiprocessing (from an .exe build from the
>> current svn) gives "ImportError: No module named _multiprocessing" and
>> the test suite skips the test for the same reason.
> 
> Can you tell why it's not building?
I was hoping you wouldn't ask :) I can't run VS visually at the
moment so I'm down to inspecting the .sln file by hand which
I have no experience of. I suspect that someone better versed
in VS solution files than I needs to add the module in. I'll try to
get to it unless someone else chips in.

Here is a diff for the tests under py3k, I applied the svnmerge, then 
ben's patch and made a couple of import changes.

I'm sorry; I've had a look and there's no chance of my updating the
solution and project files by hand; and I can't run Visual Studio at the
moment. I'll try emailing Trent or Christian in the hope that one of
them's available to do it.

Trent's supplied me with enough info to patch the project files
manually. The attached patch against r64120 results in the
_multiprocessing module building. I'm running the tests now but I'll
upload the patch in any case.

Thanks very much Tim and Trent. I've merged multiprocessing into Py3k
and added the Windows files. I hope that works.

Debian PPC build crash:
http://python.org/dev/buildbot/stable/ppc%20Debian%20unstable%203.0/buil
ds/1063/step-test/0
...snip
Re-running test 'test_multiprocessing' in verbose mode
test test_multiprocessing crashed -- <class 'OSError'>: [Errno 38] 
Function not implemented
Traceback (most recent call last):
 File "./Lib/test/regrtest.py", line 601, in runtest_inner
 indirect_test()
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/test/test_multiprocessing.py", line 1764, in test_main
 ProcessesMixin.pool = multiprocessing.Pool(4)
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/__init__.py", line 225, in Pool
 return Pool(processes, initializer, initargs)
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 84, in __init__
 self._setup_queues()
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/pool.py", line 131, in _setup_queues
 self._inqueue = SimpleQueue()
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/queues.py", line 315, in __init__
 self._rlock = Lock()
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 106, in __init__
 SemLock.__init__(self, SEMAPHORE, 1, 1)
 File "/home/pybot/buildarea/3.0.klose-debian-
ppc/build/Lib/multiprocessing/synchronize.py", line 38, in __init__
 sl = self._semlock = _multiprocessing.SemLock(kind, value, maxvalue)
OSError: [Errno 38] Function not implemented
[801671 refs]
make: *** [buildbottest] Error 1
program finished with exit code 2

Small diff for the ACKs/NEWS file