Why Selenium webdriver with Python can't reach to a website

Question

I am trying to use selenium with chrome driver to connect to a website. But it couldn't be reached. Here is my code:

 from selenium import webdriver
 from selenium.webdriver.common.by import By
 
 CHROME_EXECUTABLE_PATH = "C://Program Files (x86)//Chrome Driver//chromedriver.exe"
 CHROME_OPTIONS = webdriver.ChromeOptions()
 CHROME_OPTIONS.add_argument("--disable-notifications")
 BASE_URL = "https://www.nordstrom.com/"
 driver = webdriver.Chrome(executable_path=CHROME_EXECUTABLE_PATH, options=CHROME_OPTIONS)
 # locators
 search_button_locator = "//a[@id='controls-keyword-search-popover']"
 search_box_locator = "//*[@id='keyword-search-input']"
 
 driver.get(BASE_URL)
 driver.find_element(By.XPATH, search_button_locator)
 driver.find_element(By.XPATH, search_box_locator).send_keys("Fave Slipper")

This code gives me some error:

E:\Python\Nordstrom.com\venv\Scripts\python.exe E:/Python/Nordstrom.com/pages/simple.py
Traceback (most recent call last):
 File "E:\Python\Nordstrom.com\pages\simple.py", line 14, in <module>
 driver.find_element(By.XPATH, search_button_locator)
 File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
 return self.execute(Command.FIND_ELEMENT, {
 File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
 self.error_handler.check_response(response)
 File "E:\Python\Nordstrom.com\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
 raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@id='controls-keyword-search-popover']"}
 (Session info: chrome=94.0.4606.61)
Process finished with exit code 1

The page looks like this:

Founded page

But the expected page should be looks like this:

Expected page

How to access this website?

• user2668284: You appear to be connecting to the site successfully, however your XPATH isn't finding what you were hoping to see
• Mohibur_Cou: I could not reach the home page. How could I find the elements I am looking for? My question is why I could not reach to the home page?
• cruisepandey: The website is directly saying To keep our site secure, we don’t allow unidentified, automated traffic. If you’d like access to our data via automation, apply to join the Nordstrom Affiliate Network! - that means you should not scrape them at all.

Answer 1

The error points out that it was unable to find the XPATH element, which is why it errored out.

The main causes for this can be either:

• the XPATH is wrong
• the element has not loaded yet on the page
• the site has detected your scraping attempt and blocked you

In this case it's a combination of the 2nd and 3rd options. Whenever you use a webdriver, it exposes javascript hooks that websites can detect. To hide your activity you should learn more on how device fingerprinting and either customize your script to hide itself or use a pre-made solution for it (such as PhantomJS).

Most likely you should also look into hiding your IP by using a proxy.

• cruisepandey: This answer is misleading, the website says To keep our site secure, we don’t allow unidentified, automated traffic. If you’d like access to our data via automation, apply to join the Nordstrom Affiliate Network!
• Mohibur_Cou: The XPATH is not wrong. How do the elements load as I couldn't reach the home page? The problem is I can't reach the page. This is because they don't allow automation. One thing more, as you said hiding my IP using a proxy. How to do that?
• Mohibur_Cou: What should be the proxy?
• Allan P: Mostly high-quality residential proxies should be fine. Yes, the xpath is correct but the site is detecting your webdriver through some fingerprinting method.
• Mohibur_Cou: What should be the proxy? I have no idea about high-quality residential proxies. Is it something like 177.202.59.58? Where could I find the proxy I should use?

Answer 2

There is a problem with your 'BASE_URL' so try another Browser to debug the issue and also try to use explicit wait before click or locate any element