Commit 0ebac35

committed

fix

1 parent ebb0bc6 commit 0ebac35Copy full SHA for 0ebac35

File tree

-1

lines changed

-1

lines changed

Lines changed: 3 additions & 1 deletion

Original file line number	Diff line number	Diff line change
`@@ -20,7 +20,7 @@ def fetch(self, url, **kwargs):`
`20`	`20`	`if response.status_code == 200:`
`21`	`21`	`response.encoding = 'utf-8'`
`22`	`22`	`return response.text`
`23`		`- except requests.ConnectionError:`
	`23`	`+ except (requests.ConnectionError, requests.ReadTimeout):`
`24`	`24`	`return`
`25`	`25`
`26`	`26`	`def process(self, html, url):`
`@@ -39,6 +39,8 @@ def crawl(self):`
`39`	`39`	`for url in self.urls:`
`40`	`40`	`logger.info(f'fetching {url}')`
`41`	`41`	`html = self.fetch(url)`
	`42`	`+ if not html:`
	`43`	`+ continue`
`42`	`44`	`time.sleep(.5)`
`43`	`45`	`yield from self.process(html, url)`
`44`	`46`	`except RetryError:`

Comments

(0)