Run shell script in python

Question

I need to execute a shell script to run the python program in via python command.

I should have to execute my python script like this

ubuntu@ip-10-32-157-231:~/hg_intcen/lib$ xvfb-run python webpage_scrapper.py http://www.google.ca/search?q=navaspot

This script need to be executed in python program since there are huge links has to be passed to that module.

I have searched to execute this shell script in python,so i used "subprocess"

The main thing is when you run this shell command it takes some time to return the result. i need the python module to execute this command as well as it has to wait for while to return the result.This is required.

I used subprocess.Popen it doesn't return the result like what i got from the bash

import subprocess
def execute_scrapping(url):
 exe_cmd = "xvfb-run python lib/webpage_scrapper.py"+" "+str(url)
 print "cmd:"+exe_cmd
 proc = subprocess.Popen(exe_cmd,shell=True,stdin=subprocess.PIPE,stdout=subprocess.PIPE)
 time.sleep(15)
 sys.stdout.flush()
 d=proc.stdout.readlines()
 return d[1]

this above is not run into exact result. Could you please suggest me to execute the bash shell command via python and get the result?

• Blender: Running Python from within Python? Ugh. Is there any reason you cannot import that script and use it as a module?
• Nava: yes ofcourse. The reason is "xvfb-run".It only executes my python module.xvfb is x virtual frame buffer.along with xvfb only that module will give the result.
• Nava: blogs.uni-osnabrueck.de/rotapken/2008/12/03/…
• unutbu: You might be able to replace xvfb-run with PyVirtualDisplay. Then you could import webpage_scrapper.py instead of using subprocess.
• Nava: @unutbu It is not working with pyvirtualdisplay also..that xvfb command only doing that Is that any error? >>> from pyvirtualdisplay import Display >>> Display(visible=1, size=(320, 240)).start() >>> crawl(url) manage.py: cannot connect to X server :1188

Answer 1

Try:

proc.wait()

instead of your time.sleep(15) call.

From the docs:

Popen.wait() - Wait for child process to terminate. Set and return returncode attribute.

Answer 2

You should use the communicate() method to wait that the external process completes.

stddata, stderr = proc.communicate()

If you have to exchange messages between the two process then look into the pexpect module:

From the website:

 import pexpect
 child = pexpect.spawn ('ftp ftp.openbsd.org')
 child.expect ('Name .*: ')
 child.sendline ('anonymous')
 child.expect ('Password:')
 child.sendline ('[email protected]')
 child.expect ('ftp> ')
 child.sendline ('cd pub')
 child.expect('ftp> ')
 child.sendline ('get ls-lR.gz')
 child.expect('ftp> ')
 child.sendline ('bye')