Submitting a form using Requests Python

Question

So I want to automate the filling in and submitting of a form and I am using Requests to do that.

From inspecting elements I know the url to submit to and the type of submition (post):

method="post"
action="/sformsubmit"
enctype="multipart/form-data"

My problem is that my request is not going through and being fairly new to this I am not sure why.

On my webpage I have two buttons side by side like so :

 ___________________________ ________________________________
 | Submit decleration | | Reset Form |
 ___________________________ ________________________________

And when I inspect elements on that line i get:

<td align="center" colspan="2">
 <input type="hidden" name="inLeader" value>
 <input type="hidden" name="inMember" value>
 <input type="hidden" name="version" value="0">
 <input type="hidden" name="key" value="2013:a:c:3:2s">
 <input type="submit" value="Submit declaration">
 <input type="reset" value="Reset form">
</td>

I am trying the following:

>>> payload = {'inLeader':'', 'inMember':'', 'version':'0', 'key':'2013:a:c:3:2s'}

However it doesnt seem to work and isnt generating any errors.

Any help would be grateful. Thanks in advance

• aychedee: What error are you getting? What is the response? Your payload is incorrect but it is still hard to answer your question properly without knowing what the actual response is.
• user2897415: Im not getting any errors, it seems to go through fine just nothing happens. The status_code when I check via p.status_code = 200
• Nafiul Islam: Traceback? Also, the URL you are trying to post to would be nice to have.
• user2897415: @GamesBrainiac the URL is a private link where we submit assignments too, it wouldnt be accessible without a username and password

Answer 1

Okay, your payload is wrong. BUT. I'm not sure if changing it will actually help because you didn't include the error message you are receiving.

payload = {
 'inLeader':'',
 'inMember':'',
 'version':'0',
 'key':'2013:a:c:3:2s',
}

What you need to understand about HTML forms and POST requests is that when you click a submit button on a form it sends the value attribute of any field with a name attribute. The input field with the type submit doesn't get sent for example. It has no name. I'm suspicious that the inLeader and inMember fields have no data. Is this being set via Javascript somehow?

You mention in a comment that you need to be logged in order to access the form? This most likely means that you also need to send the correct cookie along with the request. So, visiting the URL I get asked for a username/password. This website is using basic auth.

requests supports this. Example below:

import requests
from requests.auth import HTTPBasicAuth
requests.get(url, auth=HTTPBasicAuth('your username', 'your password'))

Try just making a get request and seeing if you can at least get a 200 response. That means that the auth is working. Then you can try and do the actual post.

• user2897415: I dont think it is being set via the Javascript, I have access to another form that has been sent so the <input type="submit" value="Submit declaration"> has been changed to <input type="submit" value="Change declaration"> and in that one they are still empty
• aychedee: Really? What you actually need in order to debug this properly is not the source code. But an actual post request. You can see these in Chrome under the network tab of the chrome dev tools. When you submit the form manually it will record the POST and you can see what information has been sent. The URL looks fine to me.
• user2897415: where should I be looking in the network tab to see what is being sent ... I have clicked on the request and in the preview tab it shows me the html : what I see in inspect element so what part shows me what is being sent
• aychedee: You click on the specific http request and view the headers.
• user2897415: thanks got it working, the problem was not using BasicAuth - cheers!

Answer 2

You can use mechanical soup!

import mechanicalsoup
browser = mechanicalsoup.StatefulBrowser()
browser.open("https://YOURSITE.example") #<Response [200]>
browser.select_form('form CSS selector') #Right Click on element in inspect and select copy as CSS selector
browser["inLeader"] = ""
browser["inMember"] = ""
#and more!
response = browser.submit_selected()

Learn more : https://mechanicalsoup.readthedocs.io/en/stable/tutorial.html

Good Job!