Running multiple Python scripts

Question

I would like to create a simple Python program that will concurrently execute 2 independent scripts. For now, the two scripts just print a sequence of numbers but my intention is to use this program to concurrently run a few Twitter streaming programs in the future.

I suspect I need to use subprocess.Popen but I cannot quite get my head around what arguments I should put in there. There was a similar question on StackOverflow but the code provided there (pasted below) doesn't print anything. I will appreciate your help.

My files are:

thread1.py
thread2.py

import subprocess
subprocess.Popen(['screen', './thread1.py']))
subprocess.Popen(['screen', './thread2.py'])

• wanderlust: Isn't it simpler and more obvious to run parallel processes using pipelines in Unix? I.e. ./thread1.py;./thread2.py
• Aashish P: Yes, normal Unix process pipeline will do the trick. Whereas just a heads up, if the operations are i/o bound you should consider using threads over processes in Python.
• user3722736: Thanks, I've tried what you suggested: python ./thread1.py;./thread2.py but I'm only getting the printed numbers from the first script and I would like the two scripts to be running at the same time.
• wanderlust: Because of the pipeline usage you should run both your scripts as separate python programs: python thread1.py;python thread2.py. In your case I assume you don't have the python preambula, so your scripts require to use python interpreter, as I stated above. That's the reason, why you second script didn't run. Pay attention you should be in the directory your files are located in, and you don't need the "./" prefix, that indicates you to take file from the present folder.
• user3722736: I've tried to use the above Unix method for two scripts that download Twitter timelines. Unfortunately, it appears that the scripts are not run in parallel. Each of my scripts builds a separate MySQL table and I can see that the table for the second script only starts building after I interrupt the first script. Is this the case that @AashishP mentioned and should I be using threads?

Answer 1

Use supervisord

supervisord is process control system just for the purpose of running multiple command line scripts.

It features:

• multiple controlled processes
• autorestarting failed runs
• log stdout and stderr output
• starting scripts in order (using priority)
• command line utility to view latest log output, stop, start, restart the processes

This solution works only on *nix based systems, it is not available on Windows.

Answer 2

As wanderlust mentioned, why do you want to do it this way and not via linux command line?

Otherwise, the solution you post is doing what it is meant to, i.e, you are doing this at the command line:

screen ./thread1.py
screen ./thread2.py

This will open a screen session and run the program and output within this screen session, such that you will not see the output on your terminal directly. To trouble shoot your output, just execute the scripts without the screen call:

import subprocess
subprocess.Popen(['./thread1.py'])
subprocess.Popen(['./thread2.py'])

Content of thread1.py:

#!/usr/bin/env python
def countToTen():
 for i in range(10):
 print i
countToTen()

Content of thread2.py:

#!/usr/bin/env python
def countToHundreds():
for i in range(10):
 print i*100
countToHundreds()

Then don't forget to do this on the command line:

chmod u+x thread*.py

• user3722736: Thanks, I've tried using the above script (in IDLE) but I got many errors. Traceback (most recent call last): line 3, in <module> subprocess.Popen(['./thread1.py']) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 709, in init errread, errwrite) File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1326, in _execute_child raise child_exception OSError: [Errno 13] Permission denied
• jonnybazookatone: This is most likely related to the content of your 'thread1' and 'thread2'. Can you give any details?
• user3722736: Sure, I got the UNIX solution to work but I'm curious what is wrong with my subprocess.Popen. Here is thread1: 'def countToTen(): for i in range(10): print i countToTen()' and thread2:'def countHundreds(): for i in range (10): print i*100 countHundreds()' I'm getting errors when I run this code: 'import subprocess subprocess.Popen(['thread1.py']) subprocess.Popen(['thread2.py'])' P.S Sorry for my bad formatting, I need to learn how to format here
• jonnybazookatone: You are missing the './' infront of thread1.py and thread2.py. Also, within each of the files, thread1.py and thread2.py you will need to point to the python executable, i.e. '#!/usr/bin/env python'. You also need to make them executable, i.e., 'chmod u+x thread*.py'. Otherwise it should work as you want. I will edit my post to include this.
• user3722736: Thank you, it works now from command line (but not if I just try to run it from IDLE but perhaps you can't run concurrent programs there). I need to read up on the "chmod" line but this was a very valuable lesson for a Python beginner.

Answer 3

You can also just open several Command Prompt windows to run several Python programs at once - just run one in each of them:

In each Command Prompt window, go to the correct directory (such as C:/Python27) and then type 'python YourCodeNo1.py' in one Command Prompt window, 'python YourCodeNo2.py' in the next one ect. .

I'm currently running 3 codes at one time in this way, without slowing any of them down.