using python to post data to a form

Fri Apr 8 18:39:02 EDT 2011

On 04/04/2011 01:01 PM, Corey Richardson wrote:
> On 04/04/2011 01:36 AM, Littlefield, Tyler wrote:
>> Hello:
>> I have some data that needs to be fed through a html form to get
>> validated and processed and the like. How can I use python to send data
>> through that form, given a specific url? the form says it uses post, but
>> I"m not really sure what the difference is. would it just be:
>> http://mysite.com/bla.php?foo=bar&bar=foo?
>> If so, how do I do that with python?
>>> import urllib
> import urllib2
>> url = "http://www.foo.com/"
> data = {"name": "Guido", "status": "BDFL"}
>> data = urllib.urlencode(data)
> request = urllib2.Request(url, data)
> response = urllib2.urlopen(request)
>> page = response.read()
>> So yeah, passing in a Request object to urlopen that has some
> urlencode'ed data in it.

Real life example:
I query for bugs:
def make_form_data(query=None):
 """Factory function to create a post form query to submit on a html 
webpage.
 @param query - the query string from the existing query list of the 
query webpage.
 """
 return {
 'init' : EMPTY,
 'NextForm' : EMPTY,
 'LastForm' : FORM_QUERY,
 'ACTION' : ACTION,
 'class' : CLASS,
 'personalQuery' : query,
 'sharedQuery' : EMPTY,
 '.cgifields' : CONFIG_QUERY,
 '.cgifields' : SHARED_QUERY
 }
def authentication_setup(username=None, password=None, url=None):
 """Setup an authentication for a super-url (root url) on a given 
securised web server.
 @param username - String
 @param password - String
 @param url - String
 """
 # Password Manager creation
 pwd_manager = urllib2.HTTPPasswordMgrWithDefaultRealm()
 # As we set the first parameter to None
 # the Password Manager will always use
 # the same combination username/password
 # for the urls for which 'url' is a super-url.
 pwd_manager.add_password(None, url, username, password)
 # Authentication Handler creation from the Password Manager.
 auth_handler = urllib2.HTTPBasicAuthHandler(pwd_manager)
 # Opener creation from the Authentication Handler.
 opener = urllib2.build_opener(auth_handler)
 # Tous les appels a urllib2.urlopen vont maintenant utiliser le handler
 # Ne pas mettre le protocole l'URL, ou
 # HTTPPasswordMgrWithDefaultRealm sera perturbe.
 # Vous devez (bien sur) l'utiliser quand vous recuperez la page.
 urllib2.install_opener(opener)
 # l'authentification est maintenant geree automatiquement pour nous
def post_request(url=None, data=None, headers=None):
 """Post a request form on a given url web server.
 @param url - String
 @param data - Dictionnary
 @param headers - Dictionnary
 @return response The web page (file) object.
 """
 if headers is None:
 headers = {
 'User-Agent' : __file__,
 'Content-Type' : 'application/x-www-form-urlencoded',
 'Accept' : 'text/html'
 }
 query = urllib.urlencode(data) if data is not None else ''
 request = urllib2.Request(url, query, headers)
 response = urllib2.urlopen(request)
 #print('INFO', response.info())
 return response
AND MAIN CODE:
 try:
 webpage = post_request(url=MAIN_URL, data=make_form_data(query))
 html = webpage.read()
 print('Authentication Information: access granted.')
except URLError, e:
 print('Authentication Information: {msg}.'.format(msg=e))
 sys.exit(1)
That's all folks Authentication+Posting a request.
make_form_data() is the most important.
To find cgi data dict you can use ClientForm.py (google it!) it is a 
good helper to find form data.
Regards
Karim