homepage

See my question at http://stackoverflow.com/questions/8785899/hang-in-python-script-using-sqlalchemy-and-multiprocessing
I can't improve on the analysis by Lorenzo Bolla,
so I reproduce his example below. This example hangs if
BadExc is thrown, but not if GoodExc is thrown.
The only difference between these is that the GoodExc does
not require an argument be passed to its constructor, while
BadExc does.
This looks like a bug to me, though I suppose it might
be some pickling limitation.
I have confirmed this behavior is present in 2.6.6, 2.7.2,
and 3.1.3, all tested on Debian squeeze.
 Regards, Faheem
#################################################
import multiprocessing
class BadExc(Exception):
 def __init__(self, a):
 '''Non-optional param in the constructor.'''
 self.a = a
class GoodExc(Exception):
 def __init__(self, a=None):
 '''Optional param in the constructor.'''
 self.a = a
def do(kwargs):
 i = kwargs['i']
 print i
 raise BadExc('a')
 # raise GoodExc('a')
 return i
pool = multiprocessing.Pool(processes=5)
results = []
arglist = []
for i in range(10):
 arglist.append({'i':i})
r = pool.map_async(do, arglist, callback=results.append)
try:
 # set a timeout in order to be able to catch C-c
 r.get(1e100)
except KeyboardInterrupt:
 pass
print results

This is not specific to multiprocessing. It is really an issue with the pickling of exceptions:
 >>> import cPickle
 >>> class BadExc(Exception):
 ... def __init__(self, a):
 ... '''Non-optional param in the constructor.'''
 ... self.a = a
 ...
 >>> a = cPickle.dumps(BadExc(1))
 >>> cPickle.loads(a)
 Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 TypeError: ('__init__() takes exactly 2 arguments (1 given)', <class '__main__.BadExc'>, ())
I think that when you create a new exception class with an __init__() method, you need to make sure that self.args is set properly by calling the __init__() method of the parent class using the same arguments. So you can instead do
 class BadExc(Exception):
 def __init__(self, a):
 '''Non-optional param in the constructor.'''
 Exception.__init__(self, a)
 self.a = a

Thanks to sbt for his helpful and clear explanation.
The following bug report appears relevant, http://bugs.python.org/issue1692335. It seems the proposed fix
was never applied because it caused breakage to existing gcode.
It is not clear to me whether this behavior is considered a bug
or a feature. :-)

2.6 and 3.1 only get security fixes.
I am only guessing that this is still an issue for 3.2. A test with 3.2.2 would be good. If this is really about pickling behavior w/r/t exceptions that cannot be changed then this should be closed.

This is an issue with SQLAlchemy exceptions, and has been worked around by Mike Bayer in http://www.sqlalchemy.org/trac/ticket/2371
For the record, I think the real problem is that Python exception pickling is broken, see http://bugs.python.org/issue1692335
It would be nice to see this fixed, otherwise this issue will continue to plague the Python standard libraries as well as other libraries. Library writers don't seem to be aware of the issue.

This is a duplicate of #9244 and #9400 which have been fixed by wrapping unpicklable exceptions in picklable exceptions.
The larger issue of many exception classes being unpicklable, is dealt with in #1692335.