YouTube Search Result Scraper

Question

This is a program I wrote in Python using the BeautifulSoup library. The program scrapes YouTube search results for a given query and extracts data from the channels returned in the search results.

I'm just looking for some tips on how to make my code look (and function) better. I removed most of the redundancies but to me the code still feels ugly.

Suggestions?

#!/usr/bin/python
# http://docs.python-requests.org/en/latest/user/quickstart/
# http://www.crummy.com/software/BeautifulSoup/bs4/doc/
import csv
import re
import requests
import time
from bs4 import BeautifulSoup
# scrapes the title 
def getTitle():
 d = soup.find_all("h1", "branded-page-header-title")
 for i in d:
 name = i.text.strip().replace('\n',' ').replace(',','').encode("utf-8")
 f.write(name+',')
 print('\t\t%s') % (name)
# scrapes the subscriber and view count
def getStats():
 b = soup.find_all("li", "about-stat ") # trailing space is required.
 for i in b:
 value = i.b.text.strip().replace(',','')
 name = i.b.next_sibling.strip().replace(',','')
 f.write(value+',')
 print('\t\t%s = %s') % (name, value)
# scrapes the description
def getDescription():
 c = soup.find_all("div", "about-description")
 for i in c:
 description = i.text.strip().replace('\n',' ').replace(',','').encode("utf-8")
 f.write(description+',')
 #print('\t\t%s') % (description)
# scrapes all the external links 
def getLinks():
 a = soup.find_all("a", "about-channel-link ") # trailing space is required.
 for i in a:
 url = i.get('href')
 f.write(url+',')
 print('\t\t%s') % (url)
# scrapes the related channels
def getRelated():
 s = soup.find_all("h3", "yt-lockup-title")
 for i in s:
 t = i.find_all(href=re.compile("user"))
 for i in t:
 url = 'https://www.youtube.com'+i.get('href')
 rCSV.write(url+'\n')
 print('\t\t%s,%s') % (i.text, url) 
f = open("youtube-scrape-data.csv", "w+")
rCSV = open("related-channels.csv", "w+")
visited = []
base = "https://www.youtube.com/results?search_query="
q = ['search+query+here']
page = "&page="
count = 1
pagesToScrape = 20
for query in q:
 while count <= pagesToScrape:
 scrapeURL = base + str(query) + page + str(count)
 print('Scraping %s\n') %(scrapeURL)
 r = requests.get(scrapeURL)
 soup = BeautifulSoup(r.text)
 users = soup.find_all("div", "yt-lockup-byline")
 for each in users:
 a = each.find_all(href=re.compile("user"))
 for i in a:
 url = 'https://www.youtube.com'+i.get('href')+'/about'
 if url in visited:
 print('\t%s has already been scraped\n\n') %(url)
 else:
 r = requests.get(url)
 soup = BeautifulSoup(r.text)
 f.write(url+',')
 print('\t%s') % (url)
 getTitle()
 getStats()
 getDescription()
 getLinks()
 getRelated()
 f.write('\n') 
 print('\n')
 visited.append(url)
 time.sleep(3)
 count += 1 
 time.sleep(3)
 print('\n')
 count = 1
 print('\n') 
f.close()

Answer 1

I'm pretty much a n00b to programming myself, so take my advice with a grain of salt... but I would try making each of your "get..." functions into a method of a class (let's say YoutubeVid). It's __init__ would set all the attributes at once, without printing. A seperate function, let's say print_attributes could do the printing. Once you code that part, you would replace this:

 else:
 r = requests.get(url)
 soup = BeautifulSoup(r.text)
 f.write(url+',')
 print('\t%s') % (url)
 getTitle()
 getStats()
 getDescription()
 getLinks()
 getRelated()
 f.write('\n') 
 print('\n')
 visited.append(url)
 time.sleep(3)

With something like this:

 else:
 r = requests.get(url)
 soup = BeautifulSoup(r.text)
 video_page = YoutubeVid(soup)
 print_attributes(video_page)

I'm sorry I don't have the time to work out a more detailed example, but if that makes any sense to you, maybe you can give it a try and post what you come up with.

Also, a minor detail regarding function names... Mixed case like getTitle() is depricated. Lowercase with underscores like get_title() is prefered. See the PEP Style Guide.

Answer 2

Code Style

There are multiple PEP8 code style violations, some of them are:

• naming - use lower_case_with_underscores naming style
• spaces around operators
• blank lines between imports and functions

You should also improve your variable naming - for example, d, b, i are not meaningful - think of more descriptive names - remember: Code is much more often read than written.

HTML-parsing and Web-scraping

• instantiate requests.Session() and reuse to make requests - this would give you a performance boost "for free":

  if you're making several requests to the same host, the underlying TCP connection will be reused, which can result in a significant performance increase

• you can replace .find_all() calls with more explicit and robust select() calls and use CSS selectors. For instance, soup.find_all("h1", "branded-page-header-title") would become soup.select("h1.branded-page-header-title")

• it is also a good idea to specify a parse that BeautifulSoup uses under-the-hood explicitly:

  soup = BeautifulSoup(r.text, "html.parser") # built-in, no extra dependencies
  # soup = BeautifulSoup(r.text, "lxml") # the fastest
  # soup = BeautifulSoup(r.text, "html5lib") # the most lenient
  

Other

• use with context manager when dealing with file-like objects
• either remove the unused csv import or use it to write the results into CSV files
• convert comments before the functions into proper docstrings