Finding solutions on GitHub and Stack Overflow

Question

The first part of the code opens Chrome and navigates to GitHub. If I don't find my solution, I open a new tab with ctrlt. Then my code goes to Stack Overflow and searches there for solutions.

from selenium import webdriver
from selenium.webdriver.common.keys import *
import time
import keyboard
q = input("what is your programming question : ")+(' python')
class github :
 path = "c:/users/admin/appdata/local/programs/python/python38-32/chromedriver.exe"
 driver = webdriver.Chrome(path)
 driver.get("https://github.com/")
 question = driver.find_element_by_name("q")
 time.sleep(3)
 question.send_keys(q)
 question.send_keys(Keys.RETURN)
 while True:
 if keyboard.is_pressed("ctrl+t"):
 driver.switch_to_window(driver.window_handles[-1])
 driver.get("https://stackoverflow.com/")
 question_2 = driver.find_element_by_name("q")
 question_2.send_keys(q)
 question_2.send_keys(Keys.RETURN)
 break

• aki: Did you consider using URL like https://stackoverflow.com/search?q=test for the search, instead of finding DOM element, and entering keystrokes ? You can add filters of your own, like tags https://stackoverflow.com/search?q=+test+%5Bpython%5D where %5B is [ and %5D is ]
• Ben A: I would argue that the class isn't necessary here, since this seems like a small script.
• bliboy: @Linny I know that the class wasn't necessary. I wanted to be more comfortabel using classes and to get to know it better.
• Vishesh Mangla: Scrapy should be your choice if you are interested in using classes. In Python, it's not like Java where you have to put class everywhere. Modular code with functions is good enough.

Answer 1

Selenium

Selenium imposes a lot of overhead and complexity that you don't need to deal with. If you needed to scrape, use requests + beautifulsoup; but you don't: StackExchange has a good API, so you should strongly prefer using that. It will get you results more efficiently and the results will be more accurate and less fragile.

PEP8

class github :

should be

class Github:

Answer 2

For simply content scraping without JavaScript and ajax content try scrapy for best practices. Scrapy uses Python classes by default as it is a Python framework.

Easy tutorial to learn Scrapy:

Scrapy Tutorial on Youtube

Official site of ScrappingHub

Selenium is good for scraping dynamic content and causes unnecessary overhead as mentioned in above answer.

For above code: Try avoiding time.sleep and use EC.presence_of_element_located and similar functions to obtain desired behavior. Selenium Waits