How to scrape javascript webpage using python standard libs only

Question

I have to scrape a website that uses javascript to display content. I have to use standard libs only as I will run this script on a server where there is not any browser. I have found selenium but it requires a browser that in my case is not possible to install.

Any idea or solution?

• narko: Why don't you rely on Scrapy for doing the task? Avoid reinventing the wheel.
• Vikas Ojha: You can use Requests library.
• Tushar Gupta: Scarpy , Beautifulsoup are pretty good libraries for the same
• Hafiz Muhammad Shafiq: These modules (Requests,Beautifulsoup) could not do it
• pmccallum: @Shafiq Do you mind if I ask why requests and bs4 couldn't complete the task? These would have been my first go-to solutions.

Answer 1

Have a look at Ghost.py http://jeanphix.me/Ghost.py/. It doesn't require a browser.

pip install Ghost.py
from ghost import Ghost
ghost = Ghost()
page, resources = ghost.open('http://stackoverflow.com/')

Answer 2

You didn't mention anything about how the website is using javascript, but if it uses AJAX requests that are triggered after any kind of user interaction, you will need to use something like Selenium to automatize that behaviour. Here, you can find a short tutorial of how to scrape with Scrapy + Selenium. This of course requires a browser previously installed in your machine.