Python multiprocessing question

Question

Trying to think of the best way to code 2 processes that have to run in parallel. I not even sure if multiprocessing is the preferred module.

I am generating a lot of data for a long period of time with a dataCollector, but I would like to check the data periodically with a dataChecker while the dataCollector keeps running. In my mind there are 2 significant times that I consider, one, the time in which the dataCollector dumps a file a begins writing another, which is the same time that the dataChecker will start analyzing the dumped file, and two, the time in which the dataChecker is finished and begins waiting for the dataCollector again.

Can someone suggest a general outline with the multiprocessing module? Should I be using a different module? Thanks

Answer 1

Why would you use any module at all? This is simple to do by having two separate processes that start at the same time. The dataChecker would list all files in a directory, count them, and sleep for a short time (several seconds or more). Then it would do it again and if the number of files changes, it opens the new ones, reads them and processes them.

The synchronisation of the two processes would be done entirely through mailboxes, implemented as a directory with files in it. A message is received only when the dataCollector starts writing a new message.

• Clueless: That sounds overly complicated. Why not just pipe dataCollector into dataChecker, and have it write to stdout when there is data to be checked?
• agf: I think it's simpler. Why add code complexity by making the processes aware of each other? This answer is absolutely right, it's the data availability the checker cares about, not the state of the collector, and the collector doesn't care about the checker at all.