Scraping javascript rendered HTML page in python

You can Try PhantomJs or Casperjs

There are more node wrappers written over phantom and casperjs one of the most efficient and scalable is "ghost town"

One approach that may not be the fastest, but is most likely to succeed, is to use Selenium. The following function should do the job: Given an URL that holds javascript generated content, retrieve the dynmaic website and return its rendered html. Note that instead of Chrome you can use any other supported browser (e.g., Firefox, Safari or IE). Have a look at the docs:

https://www.selenium.dev/selenium/docs/api/py/api.html#

def retrieve_html_from_js_website(url, path_to_chrome_binary, threshold_waiting_time=4):
 import time
 from selenium import webdriver
 from selenium.webdriver.chrome.service import Service
 user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'
 options = webdriver.ChromeOptions()
 options.add_argument(f'user-agent=[{user_agent}]')
 options.add_argument('--disable-blink-features=AutomationControlled')
 options.add_experimental_option("excludeSwitches", ["enable-automation"])
 options.add_experimental_option("detach", True)
 with webdriver.Chrome(service=Service(path_to_chrome_binary), options=options) as driver:
 # Note that there are many creative websites that use mechanisms 
 # to prevent browsers instantiated with Selenium from crawling 
 # their content. Some mechanisms are listed in the following:
 # https://piprogramming.org/articles/How-to-make-Selenium-undetectable-and-stealth--7-Ways-to-hide-your-Bot-Automation-from-Detection-0000000017.html
 driver.get(url)
 time.sleep(threshold_waiting_time)
 return driver.page_source

From here you can perform any parsing operation, such as extracting javascript generated URLs. For this particular task, I prefer using Beautiful Soup, although Selenium can do the job as well.

https://beautiful-soup-4.readthedocs.io/en/latest/