Input html form data from python script

Question

I am working on a project and I need to validate a piece of data using a third party site. I wrote a python script using the lxml package that successfully checks if a specific piece of data is valid.

Unfortunately, the site does not have a convenient url scheme for their data and therefor I can not predict the specific url that will contain the data for each unique request. Instead the third party site has a query page with a standard html text input that redirects to the proper url.

My question is this: is there a way to input a value into the html input and submit it all from my python script?

• HamZa: Take a look at that page, does it use a form with an action? If it does, then see if it uses GET or POST. Build your scrapper accordingly. If it's using JavaScript/Ajax then it can get complicated. The thing is: it's all about analysing how those pages work.

Answer 1

Yes there is.

Mechanize

Forms

List the forms

import mechanize
br = mechanize.Browser()
br.open(url)
for form in br.forms():
 print "Form name:", form.name
 print form

select form

br.select_form("form1") 
br.form = list(br.forms())[0]

login form example

br.select_form("login") 
br['login:loginUsernameField'] = user
br['login:password'] = password
br.method = "POST"
response = br.submit()

---

Selenium

Sending input

Given an element defined as:

<input type="text" name="passwd" id="passwd-id" />

you could find it using any of:

element = driver.find_element_by_id("passwd-id")
element = driver.find_element_by_name("passwd")
element = driver.find_element_by_xpath("//input[@id='passwd-id']")

You may want to enter some text into a text field:

element.send_keys("some text")

You can simulate pressing the arrow keys by using the "Keys" class:

element.send_keys("and some", Keys.ARROW_DOWN)

These are the two packages I'm aware of that can do what you've asked.