-
Notifications
You must be signed in to change notification settings - Fork 81
Backpressure when queue is full #146
Open
Description
We uncovered a case where a key customer was losing events due to the way this library is configured/built.
With the flush_interval=2 (default is 0.5), flush_at=100 and max_queue_size=1000 (default 10k), their queue got full during peak hours and they started dropping events.
Their recommended solution is very reasonable:
Ideally the library would provide back pressure by blocking/waiting for acknowledgement when flushing a batch of events vs having to configure a max_queue_size.
We should implement some way to block when the queue is full, or some way to let the user know the queue size.