Check a program/script is active and running properly

Question

I can't find guidance anywhere on how, if you're running a (say python) script permanently in Raspbian on a headless RPi, to check periodically it's running ok. I've researched reporting on running scripts using ps, but feel like I'm just reinventing the wheel badly. Grateful for advice on anyone who has addressed this effectively. I'm guessing a separate python script run periodically with cron should work, but I don't know how to confirm if a specific other script is running or might have crashed but somehow remained open.. Very grateful for advice.

• geotheory: I've started a discussion in raspberrypi.org about this - there are a couple of other possible solutions there.

Answer 1

AFAIK there is no way to know if a process is running ok or just hung but still there using existing tools. I guess you have to find a way to check it, you can write down a log with the last time the process did something, then you check this log from an external program (lets use cron) and if the last time stamp is too old, you know something is wrong.

You can also make your program open a socket and respond to some sort of ping, then again from an external program you try to ping it from time to time.

If you want to try a ping-able program in python you can try this simple snippet:

# Echo server program
import socket
HOST = '' # Symbolic name meaning the local host
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #create the socket
s.bind((HOST, PORT)) #bind socket to port
s.listen(1) # start listening
conn, addr = s.accept() # if someone connects
print 'Connected by', addr
while 1:
 data = conn.recv(1024) # whenever they send data (up to 1024 bytes)
 if not data: break 
 conn.send(data) # send through the same socket what they send (make echo)
conn.close()

Then the client to ping it should look like the following (this is the program you should call from a cron job)

# Echo client program
import socket
HOST = '127.0.0.1' # The remote host (in this case local)
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.send('Hello, world') # we send some text
data = s.recv(1024) # and we should receive the same text
s.close()
print 'Received', repr(data)

Extracted from HERE

• geotheory: Making a script ping responsive sounds useful. How would this be achieved.
• javirs: @geotheory : edited my answer

Answer 2

Supervisor may be of help . You can start it at init time and declare which processes you want to monitor. Given that it's written in python you may find it easier to use than, say, daemontools.