homepage

Hi,
I was implementing a multiple process compatible logging which child processes are expected to send all messages to the parent process via a queue and then the parent process serializes and flushes them into a log file. The idea is built around multiprocessing.Queue()
The problem is the first child process is always blocked while subsequent ones can run smoothly. It seems to be blocked on semaphore_wait_signal_trap
Environment
===========
Mac OS X 10.6.5 
Python 2.6.6
Reproduction script:
====================
https://gist.github.com/818114
Command to run
====================
python log.py
Console message
===============
Dinh-Phams-MacBook-Pro:task dinhpham$ python log.py
Parent PID 90401
90403
Parent PID 90401
Parent PID 90401
90404
Parent PID 90401
90405
No child process exits
90406
Child process ID 90406
Child process ID 90404
Child process ID 90405
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
Child process PID 90404 will exits now
Child process PID 90406 will exits now
Child process PID 90405 will exits now
The child process 90406 exits
The child process 90405 exits
The child process 90404 exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
No child process exits
^C[90403]
Parent process exits
Dinh-Phams-MacBook-Pro:task dinhpham$ 
 
=========> All the process exits except the first child process one (90403)

I don't believe this is an interpreter or stdlib problem. See this post from September last year:
http://plumberjack.blogspot.com/2010/09/using-logging-with-multiprocessing.html
which shows how to use multiprocessing and logging together with rotating logs. The post includes the code for a complete working script which runs fine with Python 2.6 on Windows and Linux, but which I can't test on Mac OS X/Python 2.6 as I have no access to a Snow Leopard system.
You're welcome to run that script on your system and update this issue with your results - so I'm marking as "invalid" for now (as I don't believe there's a valid Python issue) and "pending" awaiting your feedback. If I don't hear from you in the next week or so, I'll close this issue.

I just simplified my script into a smaller one that can be easier to understand and find out what happens
https://gist.github.com/819979
Thanks

Sorry, just ignore my latest comment. It is for another issue

Hi Vinay,
Your example use a child process to receive log messages. I am implementing a thread-based solution. Do you think that multiprocessing.Queue() has major issues with threads or this problem happens only on Mac OSX implementations?
I still reproduce it quite easily on my system: Mac OS 10.6 and Python 2.6.6

I don't believe that the problem is about threads vs. processes. In the following gist:
https://gist.github.com/820132
I modified the script so that the main process uses a listener thread, rather than a listener process, to collect the logs. (Basically, I changed multiprocessing.Process to threading.Thread, changed the name of the function to listener_thread from listener_process, added an import for threading and updated the comments. No other changes were made.)
The script in the above gist runs fine on Linux Mint Debian and Windows 7 (Python 2.6 in both cases). Please run it on Snow Leopard and report your findings.

I think it may be related to this bug: http://bugs.python.org/issue6721 

Issue 6721 may well be valid - if this is a symptom of the same issue, then this issue should be closed. I'd like to know if the gist script I referred to works (or not) on Pham Cong Dinh's Snow Leopard machine.

As I've not had any response to my latest comment for over a week, I'm closing this issue, as I said I would.