Can't select element with Selenium no matter what i do

Question

I'm trying to develop a scraper using selenium but I can't select elements in the following page:

http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp

The element i trying to fetch is the search bar "Texto/Termos a serem pesquisados"

I have tried

element = driver.find_element_by_id("txtTermo")
or
element = driver.find_element_by_name("txtTermo")

I also tried xpath, and css selector.

My Code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get("http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp")

I expect that i can fetch the element so i can send keys.

But i get the following error:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"name","selector":"txtTermo"} (Session info: chrome=71.0.3578.98) (Driver info: chromedriver=2.45.615279 (12b89733300bd268cff3b78fc76cb8f3a7cc44e5),platform=Linux 4.15.0-43-generic x86_64)

• Dalvenjia: Maybe you have to wait for it, look into WebdriverWait (selenium-python.readthedocs.io/…) and ExpectedConditions (selenium-python.readthedocs.io/…)

Answer 1

The element you are trying to fetch is inside a frame. Frames are their own document and treated as a separate search context by Selenium:

>>> from selenium import webdriver
>>> driver = webdriver.Chrome()
>>> driver.get("http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp")
>>> element = driver.find_element_by_id("txtTermo")
Traceback (most recent call last):
 File "<pyshell#3>", line 1, in <module>
 element = driver.find_element_by_id("txtTermo")
 File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
 return self.find_element(by=By.ID, value=id_)
 File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
 'value': value})['value']
 File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
 self.error_handler.check_response(response)
 File "#PATH_REDACTED#\Python\Python36-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
 raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"txtTermo"}
 (Session info: chrome=71.0.3578.98)
 (Driver info: chromedriver=2.45.615291 (ec3682e3c9061c10f26ea9e5cdcf3c53f3f74387),platform=Windows NT 10.0.15063 x86_64)
>>> driver.switch_to.frame('main2')
>>> element = driver.find_element_by_id("txtTermo")
>>> element
<selenium.webdriver.remote.webelement.WebElement (session="d5a30c9f45d49a23152767da0b811f58", element="0.6642244007973428-1")>

Think of a hierarchy of frames that you can navigate through, each containing their own complete DOM hierarchy. When you are done working inside the frame, if you want to return to the top level document and do some more work, switch to the default content:

driver.switch_to.default_content()

Frames are highly discouraged these days and you aren't that likely to run into them on newer applications, but in the past I've found that (as silly as it sounds) examining the value of driver.page_source can help you get a handle on what part of the overall document you are currently acting on. This will show you only the source for the current search context, i.e. the frame you are currently switched to.

• Guilherme Amaral: Thank you dude, this problem was haunting me for a long time ! You are the best !