Pinging a list of hosts

Question

I have a simple script that will take a list of hosts and ping each host (there's about 200) a single time before moving on. This is not the most effecient method I am sure, as it is very linear. And it takes a few minutes to complete. I would ideally like to run this script every minute (so each IP address is checked every minute (the actual running of the script is controlled externally).

I was looking for some pointers on potentially making the script more effecient/dynamic.

import sys
import os
import platform
import subprocess
plat = platform.system()
scriptDir = sys.path[0]
hosts = os.path.join(scriptDir, 'hosts.txt')
hostsFile = open(hosts, "r")
lines = hostsFile.readlines()
if plat == "Windows":
 for line in lines:
 line = line.strip( )
 ping = subprocess.Popen(
 ["ping", "-n", "1", "-l", "1", "-w", "100", line],
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 out, error = ping.communicate()
 print out
 print error
if plat == "Linux":
 for line in lines:
 line = line.strip( )
 ping = subprocess.Popen(
 ["ping", "-c", "1", "-l", "1", "-s", "1", "-W", "1", line],
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 out, error = ping.communicate()
 print out
 print error
hostsFile.close()

• MHibbin: I think it may be the way I'm reading a file.
• Kev: Might want to take a look at Nagios's scheduler code to get some ideas.

Answer 1

Here's a solution using threads:

import sys
import os
import platform
import subprocess
import Queue
import threading
def worker_func(pingArgs, pending, done):
 try:
 while True:
 # Get the next address to ping.
 address = pending.get_nowait()
 ping = subprocess.Popen(ping_args + [address],
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 out, error = ping.communicate()
 # Output the result to the 'done' queue.
 done.put((out, error))
 except Queue.Empty:
 # No more addresses.
 pass
 finally:
 # Tell the main thread that a worker is about to terminate.
 done.put(None)
# The number of workers.
NUM_WORKERS = 4
plat = platform.system()
scriptDir = sys.path[0]
hosts = os.path.join(scriptDir, 'hosts.txt')
# The arguments for the 'ping', excluding the address.
if plat == "Windows":
 pingArgs = ["ping", "-n", "1", "-l", "1", "-w", "100"]
elif plat == "Linux":
 pingArgs = ["ping", "-c", "1", "-l", "1", "-s", "1", "-W", "1"]
else:
 raise ValueError("Unknown platform")
# The queue of addresses to ping.
pending = Queue.Queue()
# The queue of results.
done = Queue.Queue()
# Create all the workers.
workers = []
for _ in range(NUM_WORKERS):
 workers.append(threading.Thread(target=worker_func, args=(pingArgs, pending, done)))
# Put all the addresses into the 'pending' queue.
with open(hosts, "r") as hostsFile:
 for line in hostsFile:
 pending.put(line.strip())
# Start all the workers.
for w in workers:
 w.daemon = True
 w.start()
# Print out the results as they arrive.
numTerminated = 0
while numTerminated < NUM_WORKERS:
 result = done.get()
 if result is None:
 # A worker is about to terminate.
 numTerminated += 1
 else:
 print result[0] # out
 print result[1] # error
# Wait for all the workers to terminate.
for w in workers:
 w.join()

• MHibbin: sorry for the delayed response this part of my project has to take a massive "side-step". Couple of questions... NUM_WORKERS... I assume this is the amount of pings done in one cycle, for exaample you have "4" this would mean 4 hosts would be pinged in one go. Also am I right in assuming I would feed my ip addresses in as pending?
• MRAB: Each worker gets a host from the pending queue, sends a ping, and puts the reply in the done queue, repeating until the pending queue is empty. There are NUM_WORKERS workers, so there are at most concurrent pings. Note that it fills the pending queue before starting the workers. Another way of doing it is to replace "pending.get_nowait()" with "pending.get()" and put a sentinel such as None in the pending queue at the end to say that there'll be no more hosts; when a worker sees that sentinel, it puts it back into the pending queue for the other workers to see, and then breaks out of its loop.
• MHibbin: Ha! I didn't actually scroll down on the code!
• MHibbin: This is amazing! How many threads is too many? Is there a any calculations that I can perform to (i.e. using CPU and memory, etc) work out what the optimal number of threads is? - I have tried a few, and got the script to finish the list in about 13 seconds (it was in minutes before).
• MRAB: The optimal number of threads depends on several factors, including the ping response time, so you'll just need to test it with differing numbers of threads. I expect that increasing the number of threads will decrease the run time, but with diminishing returns, so just pick a (non-excessive) number of threads which gives you a reasonable run time.

Answer 2

It's not a complete answer, but your code would be more clear (and easy to maintain) if you won't duplicate code:

import sys
import os
import platform
import subprocess
plat = platform.system()
scriptDir = sys.path[0]
hosts = os.path.join(scriptDir, 'hosts.txt')
hostsFile = open(hosts, "r")
lines = hostsFile.readlines()
for line in lines:
 line = line.strip( )
 if plat == "Windows":
 args = ["ping", "-n", "1", "-l", "1", "-w", "100", line]
 elif plat == "Linux":
 args = ["ping", "-c", "1", "-l", "1", "-s", "1", "-W", "1", line]
 ping = subprocess.Popen(
 args,
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 out, error = ping.communicate()
 print out
 print error
hostsFile.close()

Answer 3

Why not using threads ? You could run your pings simultaneously.

This works quite well :

import sys
import os
import platform
import subprocess
import threading
plat = platform.system()
scriptDir = sys.path[0]
hosts = os.path.join(scriptDir, 'hosts.txt')
hostsFile = open(hosts, "r")
lines = hostsFile.readlines()
def ping(ip):
 if plat == "Windows":
 ping = subprocess.Popen(
 ["ping", "-n", "1", "-l", "1", "-w", "100", ip],
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 if plat == "Linux":
 ping = subprocess.Popen(
 ["ping", "-c", "1", "-l", "1", "-s", "1", "-W", "1", ip],
 stdout = subprocess.PIPE,
 stderr = subprocess.PIPE
 )
 out, error = ping.communicate()
 print out
 print error
for ip in lines:
 threading.Thread(target=ping, args=(ip,)).run()

• MHibbin: When I run this, I get the following error... OSError: [Errno 24] Too many open files
• Don Question: seems like you exceeded your file-quota. either you raise your limit or limit the amount of threads.
• MHibbin: How would I do that then?
• Mak: In the above code the file is being open, but never close that's why you are getting this error..

Answer 4

i would look into the python-nmap package.

why reinvent the wheel if there is already one free to use? You will be much more productive and flexible this way.

to check for hosts active in an ipv4 network you would usually do a:

nmap -sP 192.168.100.0/24 

which basically does a ping and takes in most cicumstances 2-5 sec.

nmap is also avaiable for windows.

• MHibbin: nmap is frowned upon in the network :/ . Plus we only want to check specific hosts not network ranges.
• Don Question: frowned? why? thats like a chirurg using a scalpel being frowned on not using a dull spoon.