Can't scrape table BeautifulSoup

Question

I'm trying to scrape the following table from this URL: https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show

This is my code:

import requests
from bs4 import BeautifulSoup
url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"
r = requests.get(url)
soup = BeautifulSoup(r.content, "lxml")
table = soup.find("table")
for row in table.findAll("tr"):
 print([i.text for i in row.findAll("td")])

However, my variable table returns None, even though there is clearly a table tag in the HTML code of the website. How do I get it?

• MZM: Navigate to the URL you provided and view the source page and I didn't see any html table tags, i.e. <table ...> <tr><td>....</td></tr></table>.
• idkwhatimdoing: If you print the output from soup in your snippet, you will see there are no <table> tags.

Answer 1

The webpage is loaded dynamically and relies on JavaScript, therefore requests won't support it. You could use another parser library such as selenium.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
url = "https://baseballsavant.mlb.com/leaderboard/outs_above_average?type=Fielder&startYear=2022&endYear=2022&split=no&team=&range=year&min=10&pos=of&roles=&viz=show"
driver.get(url)
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.TAG_NAME, 'table')))
table = driver.find_element(By.TAG_NAME, 'table')
table_html = table.get_attribute('innerHTML')
# print('table html:', table_html)
for tr_web_element in table.find_elements(By.TAG_NAME, 'tr'):
 for td_web_element in tr_web_element.find_elements(By.TAG_NAME, 'td'):
 print(td_web_element.text)
driver.close()

Or see this answer to incorporate Selenium with BeautifulSoup.