Python Web scraping

Question

I am 12 days old into Python and web scraping and managed to write my first ever automation script. Please review my code and point out blunders If any.

What do I want to achieve?

I want to scrape all chapters of each Novel in each category and post it on a WordPress blog to test. Please point out anything that I missed, and is mandatory to run this script on the WordPress blog.

from requests import get
from bs4 import BeautifulSoup
import re
r = get(site,
 headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
soup = BeautifulSoup(r.text, "lxml")
category = soup.findAll(class_="search-by-genre")
# Getting all categories
categories = []
for link in soup.findAll(href=re.compile(r'/category/\w+$')):
 print("Category:", link.text)
 category_link = link['href']
 categories.append(category_link)
# Getting all Novel Headers
for category in categories:
 r = get(category_link,
 headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
 soup = BeautifulSoup(r.text, "lxml")
 Novels_header = soup.findAll(class_="top-novel-header")
 # Getting Novels' Title and Link
 for Novel_names in Novels_header:
 print("Novel:", Novel_names.text.strip())
 Novel_link = Novel_names.find('a')['href']
 # Getting Novel's Info
 r = get(Novel_link, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
 soup = BeautifulSoup(r.text, "lxml")
 Novel_divs = soup.findAll(class_="chapter-chs")
 # Novel Chapters
 for articles in Novel_divs:
 article_ch = articles.findAll("a")
 for chapters in article_ch:
 ch = chapters["href"]
 # Getting article
 r = get(ch, headers={"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"})
 soup = BeautifulSoup(r.content, "lxml")
 title = soup.find(class_="block-title")
 print(title.text.strip())
 full_article = soup.find("div", {"class": "desc"})
 # remove ads inside the text:
 for ads in full_article.select('center, small, a'):
 ads.extract()
 print(full_article.get_text(strip=True, separator='\n'))

Answer 1

Naming

Variable names should be snake_case, and should represent what they are containing. I would also use req instead of r. The extra two characters aren't going to cause a heartache.

Constants

You have the same headers dict in four different places. I would instead define it once at the top of the file in UPPER_CASE, then just use that wherever you need headers. I would do the same for site.

List Comprehension

I would go about collecting categories in this way:

categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))]

It's shorter and utilizes a quirk in the python language. Of course, if you want to print out each one, then add this just after:

for category in categories:
 print(category)

Also, it seems like you assign category_link to the last element in the list, so that can go just outside the list comprehension.

Save your assignments

Instead of assigning the result of soup.find to a variable, then using it in a loop, simply put that soup.find in the loop. Take a look:

for articles in soup.findAll(class_="chapter-chs"):
 for chapters in articles.findAll("a"):
 ....

---

As a result of the above changes, you code would look something like this:

from requests import get
from bs4 import BeautifulSoup
import re
HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)"}
SITE = "https://readlightnovel.org/"
req = get(SITE, headers=HEADERS)
soup = BeautifulSoup(req.text, "lxml")
category = soup.findAll(class_="search-by-genre")
categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))]
category_link = categories[-1]
# Getting all Novel Headers
for category in categories:
 req = get(category_link, headers=HEADERS)
 soup = BeautifulSoup(req.text, "lxml")
 novels_header = soup.findAll(class_="top-novel-header")
 # Getting Novels' Title and Link
 for novel_names in novels_header:
 print("Novel:", novel_names.text.strip())
 novel_link = novel_names.find('a')['href']
 # Getting Novel's Info
 req = get(novel_link, headers=HEADERS)
 soup = BeautifulSoup(req.text, "lxml")
 # Novel Chapters
 for articles in soup.findAll(class_="chapter-chs"):
 for chapters in articles.findAll("a"):
 ch = chapters["href"]
 # Getting article
 req = get(ch, headers=HEADERS)
 soup = BeautifulSoup(req.content, "lxml")
 title = soup.find(class_="block-title")
 print(title.text.strip())
 full_article = soup.find("div", {"class": "desc"})
 # remove ads inside the text:
 for ads in full_article.select('center, small, a'):
 ads.extract()
 print(full_article.get_text(strip=True, separator='\n'))

Answer 2

I think you can even get rid of the regular expressions. I prefer to use the BS4 functions.

Instead of:

categories = [link['href'] for link in soup.findAll(href=re.compile(r'/category/\w+$'))]

This statement is equivalent using a CSS selector:

categories = [link['href'] for link in soup.select("a[href*=\/category\/]")]

That means: fetch all the a href tags that contain text /category/ (escaped).