Pub/Sub: Idempotent subscribers without hand-rolling it #18017

Hello everyone!

I got tired of rewriting duplicate handling in every subscriber, so I packaged it. Pub/Sub is at-least-once and the docs say to handle duplicates without saying how.

from idemkit import QueueConfig, RedisBackend
from idemkit.contrib.pubsub import pubsub_consumer
consumer = pubsub_consumer(
 backend=RedisBackend.from_url("redis://localhost:6379"),
 config=QueueConfig(max_attempts=5),
)
@consumer.handle
def process(msg) -> None:
 fulfil(msg.data)

Dedup is on the Pub/Sub message_id. A duplicate arriving while the first is still running waits and gets the same result.

One thing worth knowing either way: if your callback takes longer than the ack deadline, Pub/Sub sends the message again while the first one is still running. That's where most "why did this happen twice" reports come from.

https://github.com/idemkit/idemkit, Apache-2.0

If it's useful, I'd appreciate a star. It's new, so visibility helps a lot right now.