Selenium/Python - cannot find an element

Question

Python 3.11
Selenium 4.38

For the following URL:

https://www.amazon.es/-/en/Womens-Lace-up-Comfortable-Anti-Slip-SDMB2203W/dp/B0B5CX9VZY

On the right hand side of the page it says "Dispatches From" and "Amazon".

I'm writing a Selenium script to try to find pages where that field doesn't say Amazon, however Selenium cannot find that element in order to read it.

Class: a-size-small offer-display-feature-text-message << that space in 'small offer' causes a compound class name error, which does not resolve by inserting a period.

CSS Selector: #fulfillerInfoFeature_feature_div > div.offer-display-feature-text.a-size-small > div.offer-display-feature-text.a-spacing-none.odf-truncation-popover > span << does not find element.

XPath: '//\*\[@id="fulfillerInfoFeature_feature_div"\]/div\[2\]/div\[1\]/span' << does not find element.

Full XPath: /html/body/div\[1\]/div\[1\]/div/div\[2\]/div\[2\]/div\[2\]/div/div/div\[2\]/div\[5\]/div/div\[1\]/div/div/div/form/div/div/div/div/div\[4\]/div/div\[21\]/div/div/div\[1\]/div/div\[2\]/div\[2\]/div\[1\]/span << does not find element.

There doesn't appear to be a relevant iframe in play (there are iframes but I don't see that they apply to this element), and I don't see any shadow DOM.

How can I grab this element via Selenium?

EDIT: Request for more code. Below are many of the things I have tried:

 dispatch = driver.find_element(By.CLASS_NAME,'a-size-small offer-display-feature-text-message')
 dispatch = driver.find_element(By.CLASS_NAME,"span.a-size-small.offer-display-feature-text-message")
 dispatch = driver.find_element(By.CLASS_NAME, "'a-size-small offer-display-feature-text-message'")
 dispatch = driver.find_element(By.CSS_SELECTOR,"#fulfillerInfoFeature_feature_div > div.offer-display-feature-text.a-size-small > div.offer-display-feature-text.a-spacing-none.odf-truncation-popover > span")
 dispatch = driver.find_element(By.XPATH,"//span[contains(text(), 'Amazon')]")
 dispatch = driver.find_element(By.CSS_SELECTOR,"span.div.offer-display-feature-text.a-spacing-none.odf-truncation-popover")
 dispatch = driver.find_element(By.XPATH,'//*[@id="fulfillerInfoFeature_feature_div"]/div[2]/div[1]/span')
 dispatch = driver.find_element(By.XPATH,'//*[@id="merchantInfoFeature_feature_div"]/div[2]/div[1]/span')
 dispatch = driver.find_element(By.XPATH,"/html/body/div[1]/div[1]/div/div[2]/div[2]/div[2]/div/div/div[2]/div[5]/div/div[1]/div/div/div/form/div/div/div/div/div[4]/div/div[21]/div/div/div[1]/div/div[2]/div[2]/div[1]/span")

EDIT 2: My comments button on answers isn't working (I think it's a low karma thing) - to respond to @Corey Goldberg, I tried both of those suggestions and got the associated messages:

.a-size-small.offer-display-feature-text-message
Message: invalid selector: An invalid or illegal selector was specified
a-size-small offer-display-feature-text-message
Message: Compound class names are not allowed

• MortenB: Please post excepts from your selenium python script, so we can reproduce it. Also keep in mind Amazon look different for each users, so a screenshot where You mark out what element you cannot find with devtools may also help.
• boston_019: I've added the full line above of maybe a dozen different variations I've tried, they exceed the character limit in this comment.

Answer 1

Here are the answers to your questions. The working code is posted at the bottom.

1. Compound class name error

   The class below,

   a-size-small offer-display-feature-text-message
   

   is actually two classes. The space separates the two classes, a-size-small and offer-display-feature-text-message.

   When you use By.CLASS_NAME(), it expects a single class. If you use that entire string, it will throw, invalid selector: Compound class names not permitted. To fix this, you can either choose one of the classes to use (assuming the element can be uniquely found with only one class) or convert it to a CSS selector, .a-size-small.offer-display-feature-text-message.

   Having said that, this locator doesn't work because it actually returns 7 different elements so we need to find a better locator.

2. The CSS selector, #fulfillerInfoFeature_feature_div > div.offer-display-feature-text.a-size-small > div.offer-display-feature-text.a-spacing-none.odf-truncation-popover > span doesn't find the element

   When I tested it in the browser, it does find the element. My guess is that you aren't using a proper wait and that's why it's not found... the page is still loading in the background. A proper wait would be a WebDriverWait, as below.

   wait = WebDriverWait(driver, 10)
   wait.until(EC.visibility_of_element_located(locator))
   

   Having said that, that locator is excessively long and can be simplified.

3. The XPath, //*[@id='fulfillerInfoFeature_feature_div']/div[2]/div[1]/span doesn't find the element

   When I tested it in the browser, it works but I had to clean up all the escaping you had in the string first. The XPath string above, as written, works.

4. Absolute XPath... what you called, "full XPath"

   Don't use absolute XPaths because they are very brittle... meaning the slightest change in the DOM will break your locator.

5. IFRAME or shadow DOM

   Neither are in play here.

---

I think the main issue is that you aren't using a proper wait.

NOTE: The product you linked is showing as out of stock for me so it does not have the "Dispatches from Amazon" section. I chose another product linked from the original and it's working.

Here's what I found that works.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
url = 'https://www.amazon.es/-/en/G-STAR-Womens-Skinny-Destroyed-D19079-8971-c267/dp/B08KS8FQ6N'
driver = webdriver.Chrome()
driver.maximize_window()
driver.get(url)
wait = WebDriverWait(driver, 10)
dispatch = wait.until(EC.visibility_of_any_elements_located((By.CSS_SELECTOR, "#fulfillerInfoFeature_feature_div span")))[1].text
print(f"Dispatch info: {dispatch}")

It prints

Amazon

• jackal: I get selenium.common.exceptions.TimeoutException with this code. This is probably because the page layout varies according to the client location. So, for example, your code might work if you're in the USA. It certainly does not work here in the UK
• JeffC: Interesting... I don't doubt that might be the case. Would you send me a URL from the UK site of a product that has the "Dispatches from" section? I'll see if I can find some middle ground that will work for the US and UK sites.
• JeffC: Also note... the URL I used in my code is on the Spanish site... amazon.es. Yes, I'm in the US but it's technically the Spanish URL.
• jackal: Try this: amazon.co.uk/UGREEN-Bluetooth-Receiver-Handsfree-Connection/dp/…
• JeffC: Thanks. It works fine for me. I had to enter a UK postcode before that section had content though.

Answer 2

To find an element by class name when it has multiple classes:

i.e.: <div class="a b c"></div>

... you can use CSS selector:

.a.b.c or [class='a b c']

In your example, that would be

.a-size-small.offer-display-feature-text-message or [class='a-size-small offer-display-feature-text-message']