Submit form using requests package

Question

I know that there are many similar questions in SO, but I've tried every combination of get, post, data, params, etc in the requests package and I can not get the form to submit. I either get back an empty string, or the same page (ie: not the page I'm expecting to get after hitting Submit)

import requests
webserver = 'http://stev.oapd.inaf.it/cgi-bin/cmd'
# Returns the landing site
r = requests.get(webserver, data={'submit_form': 'Submit'}).text
# Returns an empty string
r = requests.get(webserver, params={'submit_form': 'Submit'}).text
# Returns an empty string
r = requests.post(webserver, data={'submit_form': 'Submit'}).text
# Returns the landing site
r = requests.post(webserver, params={'submit_form': 'Submit'}).text

The site changed recently, and I remember that one of these commands definitely used to work. Why is neither working now?

• blhsing: Did you add the other required form values on the page into the params sent with the request?
• Gabriel: The form is supposed to work as is, meaning it already has default parameters loaded. I did try adding more parameters, but it made no change.
• blhsing: The default values are specified only in the browser. The actual request still needs to have all the required values. Make sure all the form values are sent as params.

Answer 1

As you mentioned the website has changed recently. It appears that it is now using form-data in order to receive the POST requests and also requires the default values to be sent.

All in all, your code should look like this:

import requests
webserver = 'http://stev.oapd.inaf.it/cgi-bin/cmd'
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:66.0) Gecko/20100101 Firefox/66.0", "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8", "Accept-Language": "en-US,en;q=0.5"}
files = {
 'submit_form': (None,"Submit"),
 'cmd_version': (None,"3.2"),
 'track_parse': (None,"parsec_CAF09_v1.2S"),
 'track_colibri': (None,"parsec_CAF09_v1.2S_S35"),
 'track_postagb': (None,"no"),
 "n_inTPC": (None,"10"),
 "eta_reimers": (None,"0.2"),
 "kind_interp": (None,"1"),
 "kind_postagb": (None,"-1"),
 "photsys_file": (None,"tab_mag_odfnew/tab_mag_ubvrijhk.dat"),
 "photsys_version": (None,"YBC"),
 "dust_sourceM": (None,"dpmod60alox40"),
 "dust_sourceC": (None,"AMCSIC15"),
 "kind_mag": (None,"2"),
 "kind_dust": (None,"0"),
 "extinction_av": (None,"0.0"),
 "extinction_coeff": (None,"constant"),
 "extinction_curve": (None,"cardelli"),
 "imf_file": (None,"tab_imf/imf_kroupa_orig.dat"),
 "isoc_isagelog": (None,"0"),
 "isoc_agelow": (None,"1.0e9"),
 "isoc_ageupp": (None,"1.0e10"),
 "isoc_dage": (None,"0.0"),
 "isoc_lagelow": (None,"6.6"),
 "isoc_lageupp": (None,"10.13"),
 "isoc_dlage": (None,"0.0"),
 "isoc_ismetlog": (None,"0"),
 "isoc_zlow": (None,"0.0152"),
 "isoc_zupp": (None,"0.03"),
 "isoc_dz": (None,"0.0"),
 "isoc_metlow": (None,"-2"),
 "isoc_metupp": (None,"0.3"),
 "isoc_dmet": (None,"0.0"),
 "output_kind": (None,"0"),
 "output_kind": (None,"0"),
 "output_evstage": (None,"1"),
 "lf_maginf": (None,"-15"),
 "lf_magsup": (None,"20"),
 "lf_deltamag": (None,"0.5"),
 "sim_mtot": (None,"1.0e4"),
 ".cgifields": (None,"track_parsec"),
 ".cgifields": (None,"extinction_coeff"),
 ".cgifields": (None,"dust_sourceC"),
 ".cgifields": (None,"output_kind"),
 ".cgifields": (None,"extinction_curve"),
 ".cgifields": (None,"dust_sourceM"),
 ".cgifields": (None,"isoc_ismetlog"),
 ".cgifields": (None,"output_gzip"),
 ".cgifields": (None,"photsys_version"),
 ".cgifields": (None,"track_colibri"), 
 }
r = requests.post(webserver, headers=headers, files = files)

Hope this helps!

• Gabriel: Thank you Fozoro, this worked (although headers are not needed and there are some minor errors in the dictionary like the photsys_version element duplicated)
• Nazim Kerimbekov: I've put the header just in case. Oops, thanks for pointing it out just edited my answer.