Upper Bound on attempted_by / Dangers of Snoozing #972

Hi all!

I want to start by acknowledging that this issue resulted from our misuse of RiverQueue, not a bug in the library itself. However, I'm sharing this experience in case there's an opportunity to add safeguards against similar misuse patterns or to document them somewhere.

Context

My team uses RiverQueue for jobs that wait for incoming webhooks. When a job runs and the webhook hasn't been received yet (checked via database state), the job snoozes itself to retry later.

The Problem

We initially set a (too aggressive) 10-second snooze duration, expecting webhooks to arrive quickly after requests. This was fine initially, but caused an issue when staging was misconfigured -- jobs would snooze indefinitely in tight loops.

While endless retrying is conceptually problematic, the real impact was much worse. River appends to the attempted_by field on each execution without bounds. Our job records silently grew to enormous sizes. In a single month, we had 51TB of inter-AZ traffic costs on our AWS staging account before discovering the issue.

Ideas for Safeguards

• Limiting the size of attempted_by arrays
• A River log indicating a job has been snoozed an unreasonable amount of times

I'm also wondering: are there other implementation details in River that make endless snoozing dangerous?

cc @magaldima @themaxgoldman for vis