python requests not returning json data

Question

I would like to get the json data from for instance https://app.weathercloud.net/d0838117883#current using python requests module. I tried:

 import re
 import requests
 
 device='0838117883'
 URL='https://app.weathercloud.net'
 URL1=URL+'/d'+device
 URL2=URL+'/device/stats'
 
 headers={'Content-Type':'text/plain; charset=UTF-8',
 'Referer':URL1,
 'User-Agent':'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/48.0.2564.82 Chrome/48.0.2564.82 Safari/537.36',
 'Accept':'application/json, text/javascript,*/*'}
 
 with requests.Session() as s:
 #get html from URL1 in order to get the CSRF token
 page = s.get(URL1)
 CSRF=re.findall('WEATHERCLOUD_CSRF_TOKEN:"(.*)"},',page.text)[0]
 #create parameters for URL2, in order to get the json file
 params={'code':device,'WEATHERCLOUD_CSRF_TOKEN':CSRF}
 page_stats=requests.get(URL2,params=params,headers=headers)
 print(page_stats.url)
 print(page_stats) #<Response [200]>
 print(page_stats.text) #empty
 print(page_stats.json()) #error

But the page_stats is empty. How can I get the stats data from weathercloud?

Answer 1

Inspecting the page with DevTools, you'll find a useful endpoint: https://app.weathercloud.net/device/stats

You can "replicate" the original web request made by your browser with requests library:

import requests
cookies = {
 'PHPSESSID': '************************',
 'WEATHERCLOUD_CSRF_TOKEN':'***********************',
 '_ga': '**********',
 '_gid': '**********',
 '__gads': 'ID=**********',
 'WeathercloudCookieAgreed': 'true',
 '_gat': '1',
 'WEATHERCLOUD_RECENT_ED3C8': '*****************',
}
headers = {
 'Connection': 'keep-alive',
 'sec-ch-ua': '^\\^Google',
 'Accept': 'application/json, text/javascript, */*; q=0.01',
 'X-Requested-With': 'XMLHttpRequest',
 'sec-ch-ua-mobile': '?0',
 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
 'sec-ch-ua-platform': '^\\^Windows^\\^',
 'Sec-Fetch-Site': 'same-origin',
 'Sec-Fetch-Mode': 'cors',
 'Sec-Fetch-Dest': 'empty',
 'Referer': 'https://app.weathercloud.net/d0838117883',
 'Accept-Language': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7,es;q=0.6',
}
params = (
 ('code', '0838117883'),
 ('WEATHERCLOUD_CSRF_TOKEN', '****************'),
)
response = requests.get('https://app.weathercloud.net/device/stats', headers=headers, params=params, cookies=cookies)
# Serializing json
json_object = json.loads(response.text)

json Output:

{'last_update': 1632842172,
 'bar_current': [1632842172, 1006.2],
 'bar_day_max': [1632794772, 1013.4],
 'bar_day_min': [1632845772, 1006.2],
 'bar_month_max': [1632220572, 1028],
 'bar_month_min': [1632715572, 997.3],
 'bar_year_max': [1614418512, 1038.1],
 'bar_year_min': [1615434432, 988.1],
 'wdir_current': [1632842172, 180],
 ..............}

That's it.

• BlackMath: i edited my answer
• charlie068: Thank you! Works perfectly.