How to scrape data from website using selenium

Question

I am try to scrape data but they will provide me an error this is website link https://www.dastelefonbuch.de/Suche/Zahnarzt

from selenium import webdriver
PATH="C:\Program Files (x86)\chromedriver.exe"
url='https://www.dastelefonbuch.de/Suche/Zahnarzt'
driver =webdriver.Chrome(PATH)
driver.get(url)
vid=driver.find_element_by_xpath("//div[@class='vcard']")
for item in vid:
 title=item.find_element_by_xpath("//div[@class='name']").text
 phone=item.find_element_by_xpath("//span[@class='nr']").text
 website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
 print(title,phone,website)
 

Answer 1

Instead of vid=driver.find_element_by_xpath("//div[@class='vcard']") which returns a WebElement you need vid=driver.find_element_by_xpath("//div[@class='vcard']") which would return a list to iterate as follows:

from selenium.webdriver.common.by import By
vid = driver.find_elements(By.XPATH, "//div[@class='vcard']")
for item in vid:
 title=item.find_element_by_xpath("//div[@class='name']").text
 phone=item.find_element_by_xpath("//span[@class='nr']").text
 website=item.find_element_by_xpath("//i[@class='icon icon_url']").text
 print(title,phone,website)

• Amen Aziz: they give these error NameError: name 'find_elements_by_xpath' is not defined
• undetected Selenium: Checkout the answer update and let me know the status
• Amen Aziz: they give us wong output repeatedly print one name many time
• Amen Aziz: and also not provide website link
• undetected Selenium: @AmenAziz This answer was based on find_element_by_xpath() and find_elements_by_xpath() stuff, which I strongly believe have got you started in the right direction. repeatedly print one name sounds like a different issue all together and you may like to raise a new question.

Answer 2

When you extract xpath on an item you have to change a bit xpath, saying lookup on current node with ".//"

Change

title=item.find_element_by_xpath("//div[@class='name']").text

to

title=item.find_element_by_xpath(".//div[@class='name']").text

Checkout and feedback if it works

• Amen Aziz: it give me the correct but why put dot can you tell me
• Wonka: is cause // means global (all html), and .// means from the node (web element, your div). Also just one / means direct childrens. Here you can get more info about it and more complexity Xpaths en.wikipedia.org/wiki/XPath