Pomp changelog

Changelog History

• v0.3.0 Changes

  🚀 Does not released yet

  • 👷 BaseDownloadWorker.get_one(self, request) -> BaseDownloadWorker.process(self, request)
  • BaseDownloader.get(self, requests) -> BaseDownloader.process(self, crawler, request)
  • non BaseRequest instances are items
  • BaseHttpResponse.get_request() instead of BaseHttpResponse.request property
  • 👻 BaseMiddleware.process_(request|response|exception) now may be awaitable
  • BaseCralwer.(process|next_requests|on_processing_done) now may be awaitable
  • BaseDownloader.(start|stop) now may be awaitable
  • BaseDownloader.prepare(self) renamed to BaseDownloader.start(self,crawler)
  • BasePipeline.(start|stop|process) now may be awaitable
  • 🖨 process Future objects without wrapping it with Planned object
  • asynciotools with async/await syntax
  • removed RUNNING, CANCELLED_AND_NOTIFIED, exception method from Planned class

• v0.2.1 Changes

  🚀 Released at 2016年09月12日

  • 🛠 fix pipe start/stop/process exception processing
  • fix: AioPomp ensure_future for process_requests in main loop
  • 🖨 BaseCrawler.on_processing_done when request, middlewares, response, extract, pipelines processing was done
  • count param for BaseQueue.get_requests - number allowed concurrent requests for current downloader, may be None

• v0.2.0 Changes

  🚀 Released at 2016年03月01日

  • 🚚 middlewares logic moved to engine and BaseMiddleware.process_(request|response|exception) with crawler and downloader params
  • Downloader does not have any more middlewares param
  • BaseDownloaderMiddleware renamed to BaseMiddleware
  • BaseDownloderException renamed to BaseCrawlException
  • 👍 AioConcurrentCrawler with asyncio and concurrent futures support
  • 🚚 pomp.core.item moved to pomp.contrib.item without backward compatibility https://github.com/estin/pomp/issues/6#issuecomment-172342598
  • 👍 AioPomp with asyncio support
  • queue semaphore to prevent fetching more requests than downloader can process now
  • 🔧 bfo and dfo orders now configured via Pomp constructor
  • no recursion, internal queue is used
  • twsited is not a part of pomp
  • defer object replaced to Planned object (like Future object)

• v0.1.3 Changes

  🚀 Will not be released

  • crawler extract_items method can yield next requests
  • process exception on extract_items and next_requests by downloader middlewares
  • 👻 store result of calling sys.exc_info in exception instance

• v0.1.2 Changes

  🚀 Released at 2015年05月08日

  • ⏱ twistedtools on request timeout can raise ResponseNeverReceived or others (see _newcleint twisted api)...
  • utils.DeferredList works with deferred and not-deferred objects
  • 🛠 fixed Item object behavior (get item field value)

• v0.1.1 Changes

  🚀 Released on December 12nd 2013

  • urllib and twisted downloader on get method yield result
  • 🛠 bugfix depth first method
  • concurrenttools change try/finally to generator behavior
  • 🖨 processing requests through queue
  • 👍 better generator usage

• v0.1.0 Changes

  🚀 Released on June 14nd 2013

  • 📇 rename SimpleDownloader to UrllibDownloader
  • urllib code now in contrib/urllibtools.py
  • ENTRY_URL renamed to ENTRY_REQUESTS
  • next_url renamed to next_requests
  • 👍 async support
  • 👍 Twisted support
  • 👍 concurrent future support
  • pipelines accept crawler on start/stop/process calls
  • 👻 downloader middleware also process exception

• v0.0.2 Changes

  🚀 First public preview release.