Grab changelog

Changelog History

Page 1

• v0.6.41 Changes

  June 24, 2018

  🔄 Changed

  • 👀 Restrict pycurl version to <7.43.0.1 (see #354)

• v0.6.40 Changes

  May 13, 2018

  🛠 Fixed

  +- 🛠 Fix #346: spider does not process initial_urls
  +- 🛠 Fix #344: raise GrabInvalidUrl for pycurl error 3

• v0.6.39 Changes

  May 09, 2018

  🛠 Fixed

  • 🛠 Fix bug: task generator works incorrectly
  • 🛠 Fix bug: pypi package misses http api html file
  • 🛠 Fix bug: dictionary changed size during iteration in stat logging
  • 🛠 Fix bug: multiple errors in urllib3 transport and threaded network service
  • 🛠 Fix short names of errors in stat logging
  • 👌 Improve error handling in urrllib3 transport
  • 🛠 Fix #299: multi-added errors
  • 🛠 Fix bug: pypi package misses http api html file
  • 🛠 Fix #285: pyquery extension parses html incorrectly
  • 🛠 Fix #267: normalize handling of too many redirect error
  • 🛠 Fix #268: fix processing of utf cookies
  • 🛠 Fix #241: form_fields() fails on some HTML forms
  • 🛠 Fix normalize_unicode issue in debug post method
  • 🛠 Fix #323: urllib3 transport fails with UnicodeError on some invalid URLs
  • 🛠 Fix #31: support for multivalue form inputs
  • 🛠 Fix #328, fix #67: remove hard link between document and grab
  • 🛠 Fix #284: option headers affects content of common_headers
  • 🛠 Fix #293: processing non-latin chars in Location header
  • 🛠 Fix #324: refactor response header processing

  🔄 Changed

  • 🔨 Refactor Spider into set of async. services
  • ➕ Add certifi dependency into grab[full] setup target
  • 🛠 Fix #315: use psycopg2-binary package for postgres cache
  • Related to #206: do not use connection_reuse=False for proxy connections in spider

  ✂ Removed

  • ✂ Remove cache timeout option
  • ✂ Remove structured extension

• v0.6.38 Changes

  May 17, 2017

  🛠 Fixed

  • 🛠 Fix "error:None" in spider rps logging
  • 🛠 Fix race condition bug in task generator

  ➕ Added

  • ➕ Add original_exc attribute to GrabNetworkError (and subclasses) that points to original exception

  🔄 Changed

  • ✂ Remove IOError from the ancestors of GrabNetworkError
  • ➕ Add default values to --spider-transport and --grab-transport options of crawl script

• v0.6.37 Changes

  May 13, 2017

  ➕ Added

  • ➕ Add --spider-transport and --grab-transport options to crawl script
  • ➕ Add SOCKS5 proxy support in urllib3 transport

  🛠 Fixed

  • 🛠 Fix #237: urllib3 transport fails without pycurl installed
  • 🛠 Fix bug: incorrect spider request logging when cache is enabled
  • 🛠 Fix bug: crawl script fails while trying to process a lock key
  • 🛠 Fix bug: urllib3 transport fails while trying to throw GrabConnectionError exception
  • 🛠 Fix bug: Spider add_task method fails while trying to log invalid URL error

  ✂ Removed

  • Remove obsoleted hammer_mode and hammer_timeout config options

• v0.6.36 Changes

  February 12, 2017

  ➕ Added

  • ➕ Add pylint to default test set

  🛠 Fixed

  • 🛠 Fix #229: using deprecated response object inside Grab

  ✂ Removed

  • ✂ Remove spider project template and start_project script

• v0.6.35 Changes

  February 06, 2017

  🛠 Fixed

  • 🛠 Fix bug in deprecated grab.choose_form method
  • ➕ Add default project templates files to the distribution, by @rushter
  • 🛠 Fix #222: debug_post option fails with big post data
  • 🛠 Fix #148: pycurl ignores sigint signal

• v0.6.34 Changes

  February 04, 2017

  ➕ Added

  • ✅ Start running Grab tests in OSX environment on travis CI

  🔄 Changed

  • 📜 Use defusedxml library to parse HTML and XML, by @kevinlondon
  • Put selection, lxml and pycurl libs back to required dependencies in setup.py
  • 📚 Update installation documentation

• v0.6.33 Changes

  January 28, 2017

  ➕ Added

  • ➕ Add API documentation about few grab modules, by @rushter
  • 🏁 Start running Grab tests in Windows enviroment on appveyor CI
  • 🆕 New spider transport based on threads that allows to use Spider with any Grab network backend e.g. urllib3
  • Add remove_from_post option to grab.doc.submit method
  • ➕ Add random option to grab.change_proxy method
  • 👌 Support for deprecated attributes Spider.items and Spider.counters
  • If Spider handler raises ResponseNotValid exception, then that task goes back to task queue until task.task_try_count reaches the spider.task_try_limit

  🔄 Changed

  • 🔨 Refactor management of internal threads, fix random test failures related to cache sub-module
  • 0️⃣ Disable default logging to files while running spider by run crawl command
  • Multiple improvements in urllib3 transport
  • 0️⃣ Set default spider network & try limits to 3 (was 10)

  🛠 Fixed

  • Different bugs in urllib3 transport
  • Different bugs

  ✂ Removed

  • Remove grab.use_next_proxy method
  • ✂ Remove grab.dump method
  • ✂ Remove deprecated Spider methods and attributes

• v0.6.32 Changes

  December 31, 2017

  🛠 Fixed

  • 🛠 Fix setup.py

• 1
• 2
• Next ›
• Last »