Simple parallel multithreading script

Question

I'm trying to write a python script that creates 10 threads at a time and runs until I stop it (ie closing the console). I just need it to do basic logging, so when it fails, its just logs a failed result to a log file.

The part of the script that I need to thread is just a simple selenium script that goes to a webpage and submits a text box.

import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
browser = webdriver.Firefox()
browser.get('http://www.google.com')
search = browser.find_element_by_name('q')
search.send_keys("google search through python")
search.send_keys(Keys.RETURN) # hit return after you enter search text
time.sleep(5) # sleep for 5 seconds so you can see the results
browser.quit()

I'm unclear what is best to use in this case. As far as I can tell, Python allows concurrent tasks, but only if using the multiprocessing library. If using threading, it may be faster, but not as fast as concurrent cpu tasks.

I'm trying to implement it in futures.concurrency and would be grateful for any suggestions on the best choice to use and how to use it.

• gnat: please don't cross-post : stackoverflow.com/questions/42717454/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..."
• Ke.: ok it's deleted from there now

Answer 1

Your script seems to be IO-Bound and not CPU-Bound. As such, it will benefit from threads even having the GIL in place, because calls such as browser.get and search.send_keys are going to block until the socket sends/receives.

You'd benefit from multiprocessing if you had long running concurrent tasks that have to overcome the GIL and are constantly doing CPU work at the same time.

Use the ThreadPoolExecutor. You can change the Executor at any time to use processes, because they implement the same interface.

• Ke.: So kind of you to answer, this really is something I would have had no idea about. Thank you so much for sharing your knowledge.