Commit 85260cf

committed

Added scraper

1 parent 86580ca commit 85260cfCopy full SHA for 85260cf

File tree

+29

-1

lines changed

+29

-1

lines changed

Lines changed: 2 additions & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -26,4 +26,5 @@`
`26`	`26`	`23. LinkedIn Scraper: Scrapes user profile (v1)`
`27`	`27`	`24. Global Timer: Enter location/city name to get the associated time depending upon time zone.`
`28`	`28`	`25. Codechef Submission CLI Bot: Codechef bot to submit the solution of problems and get the result in CLI.`
`29`		`- 26. Indeed Scraper: Scrapes job listings on indeed.com and stores them in csv file.`
	`29`	`+ 26. Indeed Scraper: Scrapes job listings on indeed.com and stores them in csv file.`
	`30`	`+ 27. Wallpaper Scraper: Scrapes images from webpage and stores them in current directory`

Lines changed: 27 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -0,0 +1,27 @@`
	`1`	`+import urllib3`
	`2`	`+import shutil`
	`3`	`+from bs4 import BeautifulSoup as bs`
	`4`	`+`
	`5`	`+`
	`6`	`+def main():`
	`7`	`+`
	`8`	`+ ur = input("Input URL Here ")`
	`9`	`+`
	`10`	`+ http = urllib3.PoolManager()`
	`11`	`+`
	`12`	`+ page = http.request('GET', ur)`
	`13`	`+`
	`14`	`+ soup = bs(page.data, 'html.parser')`
	`15`	`+`
	`16`	`+ link = ["http:"+ls.get('href') for ls in soup.find_all('a', {"class" : "zoom"})]`
	`17`	`+ for vd in soup.find_all('source'):`
	`18`	`+ link.append('http:'+vd.get('src'))`
	`19`	`+ print(link)`
	`20`	`+`
	`21`	`+ for ln in link:`
	`22`	`+ index = ln.rindex('/')`
	`23`	`+ filename = ln[index+1:len(ln)]`
	`24`	`+ with http.request('GET', ln, preload_content=False)as resp, open(filename, 'wb') as out_file:`
	`25`	`+ shutil.copyfileobj(resp, out_file)`
	`26`	`+if __name__ == "__main__":`
	`27`	`+ main()`

Comments

(0)