Commit 8a9d292

committed

Initialize the dryscrape session in the middleware.

1 parent 5261f5e commit 8a9d292Copy full SHA for 8a9d292

File tree

-0

lines changed

-0

lines changed

Lines changed: 9 additions & 0 deletions

Original file line number	Diff line number	Diff line change
`@@ -8,6 +8,15 @@`
`8`	`8`	`from scrapy.downloadermiddlewares.redirect import RedirectMiddleware`
`9`	`9`
`10`	`10`	`class ThreatDefenceRedirectMiddleware(RedirectMiddleware):`
	`11`	`+ def __init__(self, settings):`
	`12`	`+ super().__init__(settings)`
	`13`	`+`
	`14`	`+ # start xvfb to support headless scraping`
	`15`	`+ if 'linux' in sys.platform:`
	`16`	`+ dryscrape.start_xvfb()`
	`17`	`+`
	`18`	`+ self.dryscrape_session = dryscrape.Session(base_url='http://zipru.to')`
	`19`	`+`
`11`	`20`	`def _redirect(self, redirected, request, spider, reason):`
`12`	`21`	`# act normally if this isn't a threat defense redirect`
`13`	`22`	`if not self.is_threat_defense_url(redirected.url):`

Comments

(0)