careful

careful is a Python library for writing resilient, well-behaved HTTP clients.

Code: https://codeberg.org/jpt/careful

Docs: https://jpt.sh/projects/careful/

PyPI - Version status-badge

Call one function to enchant an httpx.Client , making your HTTP connections more resilient and better mannered.

• Configure throttling to avoid accidental Denial-of-Service / risking getting banned.
• Retries help overcome intermittent failures on flaky sites or long crawls.
• Development caching Cache persists between runs during development, reduces redundant requests made while iterating on your crawlers & scrapers.

Example

from httpx import Client
from careful.httpx import make_careful_client
# the only function you need to call is make_careful_client
# this wraps your existing `httpx.Client` with your preferred
# careful behaviors
client = make_careful_client(
 client=Client(headers={'user-agent': 'spiderman/1.0'}),
 # retries are configurable w/ exponential back off
 retry_attempts=2,
 retry_wait_seconds=5,
 # can cache to process memory, filesystem, or SQLite
 cache_storage=MemoryCache(),
 # easy-to-configure throttling
 requests_per_minute=60,
)
# methods on client are called as they always are
# configured behaviors occur without further code changes
client.get("https://example.com")

Logo licensed from Adrien Coquet via Noun Project