Scrape website data without opening the browser (python)

Question

I want to iteratively search for 30+ items through a search button in webpage and scrape the related data.

My search items are stored in a list: vol_list

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome("driver path")
driver.get("web url")
for item in vol_list :
mc_search_box = driver.find_element_by_name("search_str")
mc_search_box.clear()
search_box.send_keys(item)
search_box.send_keys(Keys.RETURN)

After search is complete I will proceed to scrape the data for each item and store in array/list. Is it possible to repeat this process without opening browser for every item in the loop?

• erapert: See this SO answer-- especially the one by Stéphane Bruckert

Answer 1

You can't use chrome and other browsers without opening it.

In your case, headless browsers should do the job. Headless browsers simulates browser, but doesn't have GUI.

Try ghost driver/ html unit driver/ NodeJS. Then you will have to modify at least this line with the driver you want to use:

driver = webdriver.Chrome("driver path")

Good luck!

• Stelios: Thanks! I have started using PhantomJS()

Answer 2

If you're using firefox, you can apply the headless option:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
driver = webdriver.Firefox(options=options)
driver.get('your url')