homepage

> I looked at multiprocessing code, but didn't understand how to trigger a
> call to these functions. Makes it hard to come up with a unit test...
Here's a sample test:
"""
import _multiprocessing
import os
import socket
for i in range(4, 256):
 os.dup2(1, i)
s, r = socket.socketpair()
pid = os.fork()
if pid == 0:
 # child
 fd = _multiprocessing.recvfd(r.fileno())
 f = os.fdopen(fd)
 print(f.read())
 f.close()
else:
 # parent
 f = open('/etc/fstab')
 _multiprocessing.sendfd(s.fileno(), f.fileno())
 f.close()
 os.waitpid(pid, 0)
"""
What happens is that the parent process opens /etc/fstab, and sends the FD to the child process, which prints it.
Now, if I run it with the current code, here's what I get:
"""
cf@neobox:~/cpython$ ./python ~/test.py 
Traceback (most recent call last):
 File "/home/cf/test.py", line 18, in <module>
 _multiprocessing.sendfd(s.fileno(), f.fileno())
OSError: [Errno 9] Bad file descriptor
cf@neobox:~/cpython$ strace -e sendmsg ./python ~/test.py 
sendmsg(3, {msg_name(0)=NULL, msg_iov(1)=[{"10円", 1}], msg_controllen=16, {cmsg_len=16, cmsg_level=SOL_SOCKET, cmsg_type=SCM_RIGHTS, {171137285}}, msg_flags=0}, 0) = -1 EBADF (Bad file descriptor)
Traceback (most recent call last):
 File "/home/cf/test.py", line 18, in <module>
 _multiprocessing.sendfd(s.fileno(), f.fileno())
OSError: [Errno 9] Bad file descriptor
"""
Duh, it's failing with EBADF.
Why?
 cmsg->cmsg_len = CMSG_LEN(sizeof(int));
 msg.msg_controllen = cmsg->cmsg_len;
 *CMSG_DATA(cmsg) = fd;
Since we only set one byte in CMSG_DATA, if the other bytes are non-zero, the value stored in CMSG_DATA(cmsg) ends up referring to a non existing FD, hence the EBDAF.
With this simple patch:
"""
diff -r e49dcb95241f Modules/_multiprocessing/multiprocessing.c
--- a/Modules/_multiprocessing/multiprocessing.c Sun Aug 21 12:54:06 2011 +0200
+++ b/Modules/_multiprocessing/multiprocessing.c Sun Aug 21 16:56:01 2011 +0200
@@ -111,7 +111,7 @@
 cmsg->cmsg_type = SCM_RIGHTS;
 cmsg->cmsg_len = CMSG_LEN(sizeof(int));
 msg.msg_controllen = cmsg->cmsg_len;
- *CMSG_DATA(cmsg) = fd;
+ *(int *)CMSG_DATA(cmsg) = fd;
 
 Py_BEGIN_ALLOW_THREADS
 res = sendmsg(conn, &msg, 0);
@@ -154,7 +154,7 @@
 if (res < 0)
 return PyErr_SetFromErrno(PyExc_OSError);
 
- fd = *CMSG_DATA(cmsg);
+ fd = *(int *)CMSG_DATA(cmsg);
 return Py_BuildValue("i", fd);
 }
"""
It works fine.
Note that if you want to check that for FD > 255, you'd have to add something like this at the top:
for i in range(4, 256):
 os.dup2(1, i)
Note that I just used a cast to (int *) instead of memcpy() because CMSG_DATA is actually int-aligned, so there's no risk of unaligned-access, and also it's what's commonly used in the litterature.
So, would you like to add a test along those lines to test_multiprocessing?
AFAICT, multiprocessing.connection is not even documented, but this shows that it really needs some testing...