Web Scraping data using python

Question

I m just started learning web scraping using Python. My aim is to web scrape the Realtime news for Bajaj Auto Ltd. from http://money.rediff.com/companies/Bajaj-Auto-Ltd/10540026.

The problem: I'm unable to extract the contents(i.e news).

from urllib.request import urlopen
from bs4 import BeautifulSoup
url = 'http://money.rediff.com/companies/Bajaj-Auto-Ltd/10540026'
data = urlopen(url)
soup = BeautifulSoup(data)
te=soup.find('a',attrs={'target':'_jbpinter'})
lis=te.find_all_next('a',attrs={'target':'_jbpinter'})
#print(lis)
for li in lis:
 print(li.find('a').contents[0])

I m getting the error "AttributeError: 'NoneType' object has no attribute 'contents'" And I does not get the desired result.

Any input will be appreciated.

• R Nar: looks like it can't find what you think is there. try printing li and see if there is actually an a in there

Answer 1

You are trying to get the a tag twice.

Replace

for li in lis:
 print(li.find('a').contents[0])

with

for li in lis:
 print(li.get_text())

and you get this output:

Need Different Rates For Different Products: Rahul Bajaj on GST
Reforms irrespective of Bihar results: Bajaj
Auto shares in focus; Tata Motors up over 5%
We believe new Avenger will stimulate the market: Bajaj Auto's Eric Vas
BHP Billiton pins future of Indonesian coal mine on new...