Raising an exception that is unable to be unpickled causes hang in ProcessPoolExecutor

[GoogleCodeExporter]

What steps will reproduce the problem?
1. In the function submitted to a ProcessPoolExecutor, raise a custom exception 
class that takes more than one argument to __init__.

What is the expected output? What do you see instead?
I expect a call to future.result() to not hang.

What version of the product are you using? On what operating system?
I'm using ver 2.1.6 on python 2.7 on Gentoo Linux.

Please provide any additional information below.
I have attached a patch to address the issue and a test case for it.  Without 
the patch, the new test case hangs.  With the patch, it passes.

This is needed because of the issue raised in 
http://bugs.python.org/issue1692335.  An exception class that takes multiple 
arguments to __init__ can be pickled but it raises a TypeError when being 
unpickled:

In [1]: class MyError(Exception):
   ...:     def __init__(self, arg1, arg2):
   ...:         super(MyError, self).__init__(
   ...:             'arg1 = {}, arg2 = {}'.format(arg1, arg2))
   ...: 

In [2]: import pickle

In [3]: p = pickle.dumps(MyError('arg1val', 'arg2val'))

In [4]: pickle.loads(p)
---------------------------------------------------------------------------
<snip>
TypeError: __init__() takes exactly 3 arguments (2 given)

So if a child process raises an exception like this, it gets pickled and put in 
the result queue just fine.  However, in _queue_management_worker, the call to 
result_queue.get(block=True) will raise an uncaught TypeError when it tries to 
unpickle the exception.  So then the queue management just breaks.

My proposed patch attempts to catch this condition before putting the exception 
in the result queue and create a new exception that will be able to be 
unpickled but still contains information from the original exception.

Original issue reported on code.google.com by [email protected] on 30 Sep 2014 at 2:23

Attachments:

• pickling_exception.patch

[agronholm]

I would need a test for this fix too.

[srkunze]

When possible, you can workaround this by allowing this exception to have zero arguments (by default parameters). At least that fixed it for me until the patch is through.

[filmor]

The patch also contains a test, if I'm not mistaken ...

[agronholm]

Hm, yes it does! How did I miss that.

[agronholm]

I'm surprised that there is no fix for this in upstream code. Is it not a problem on Python 3?

[filmor]

It sure is. Frankly, I wasn't aware that this one here is the backports project, I'll file a bug for Python 3.3.

[agronholm]

I'll wait until upstream developers comment on it until applying any fix.

[filmor]

This is the upstream bug: http://bugs.python.org/issue24900

[agronholm]

Is this still relevant? If so, how do I reproduce it?