Commit 569d3c3

committed

Set the user agent.

1 parent f5f44b1 commit 569d3c3Copy full SHA for 569d3c3

File tree

-2

lines changed

-2

lines changed

Lines changed: 1 addition & 2 deletions

Original file line number	Diff line number	Diff line change
`@@ -15,8 +15,7 @@`
`15`	`15`	`NEWSPIDER_MODULE = 'zipru_scraper.spiders'`
`16`	`16`
`17`	`17`
`18`		`-# Crawl responsibly by identifying yourself (and your website) on the user-agent`
`19`		`-#USER_AGENT = 'zipru_scraper (+http://www.yourdomain.com)'`
	`18`	`+USER_AGENT = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'`
`20`	`19`
`21`	`20`	`# Obey robots.txt rules`
`22`	`21`	`ROBOTSTXT_OBEY = True`

Comments

(0)