Grab data using python

Question

Here website will list some data based on dropdown-filter, so i am trying to fetch those data by passing static dropdown values, but i think due to view state i am unable to grab those data.

Anyone have any idea how to grab asp.net website data which is using viewstate?

i am getting below error

Validation of viewstate MAC failed. If this application is hosted by a Web Farm or cluster, ensure that <machineKey> configuration specifies the same validationKey and validation algorithm. AutoGenerate cannot be used in a cluster.

Python Script

import requests
from bs4 import BeautifulSoup
def get_viewstate():
url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
req = requests.get(url)
data = req.text
bs = BeautifulSoup(data)
return bs.find("input", {"id": "__VIEWSTATE"}).attrs['value']
url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
data = {"__VIEWSTATE": get_viewstate(),"ST":'GJ', "ddldistrict":'AMR', "ddltaluka":'' ,"btnSearch":'Search'}
req = requests.post(url, data)
bs = BeautifulSoup(req.text)
print(bs.prettify())

• OneCricketeer: Give Selenium a try?

Answer 1

I don't think you can do this with requests but you can easily do this with selenium.

To install selenium - pip install selenium or pip3 install selenium.
Then download the latest Chromedriver from this link for your machine and copy the driver to your working directory.

You can read the selenium documentation here.

import time
from selenium import webdriver
url = "http://xlnindia.gov.in/frm_G_Cold_S_Query.aspx?ST=GJ"
browser = webdriver.Chrome()
browser.get(url)
#change this if you want to change the state from Gujrat to something else
#or you can change the state simply by changing the "?ST=GJ" part of the URL
#state = browser.find_element_by_id("ddlState")
#state.send_keys("BR")
district = browser.find_element_by_id("ddldistrict")
district.send_keys("AMR")
#Skip this if you want to include all categories into the result
category = browser.find_element_by_id("ddlCategory")
category.send_keys("R")
button = browser.find_element_by_id("btnSearch")
button.click()
time.sleep(10)
browser.save_screenshot(browser.title + ".JPEG")
html = browser.page_source
print(html)
browser.close()
browser.quit()

NOTE
If you want to use headless browser with selenium, use PhantomJS. To learn how to do this with PhantomJS read this.

• Juned Ansari: hey MD. Khairul Basar can you help me to store each field value into mysql database?
• MD. Khairul Basar: @JunedAnsari Can give it a try.
• Juned Ansari: Basar, finally i did to store data into mysqldatabase, now can you help me to fetch dropdown values dynamically, instead of specifying static value, it is based on ajax next dropdown value fills. i am trying to store whole states dropdown values in array like wise
• Juned Ansari: please check stackoverflow.com/questions/44277239/…