-
Notifications
You must be signed in to change notification settings - Fork 264
Handling WorkerLostError #409
Hi, @auvipy !
I have a question, how can we handle this scenario where a worker gets lost abruptly like with a sigkill?
In the case of a timeout, i see you raise the exception so it gets propagated upwards, but with the worker lost, it just gets caught and passes as if nothing happened to the upper method caller.
Can this be raised/what can i do?
Lines 1263 to 1273 in 81cc942
All reactions
Replies: 1 comment 3 replies
When a worker is lost, a new one should be instantiated instead.
Is that not the case?
All reactions
Hello, thanks so much for answering!
It DOES get reinstantiated, my problem is that it requeues the failed task in an infinite loop ignoring retries flag (i think because this exception doesnt get propagated) so if you have a task that always fails by lets say memory consumption, it will requeue and fail in an infinite loop.
How do you handle this scenario?
Task starts -> worker gets sigkilled -> task requeues
Task starts -> worker gets sigkilled -> task requeues
Task starts -> worker gets sigkilled -> task requeues
...
add infinitum
All reactions
This is the normal behavior for Billiard. You need to cancel the request to execute the task somehow.
If this is a Celery issue, revoking the task could do the trick.
All reactions
-
❤️ 1
Thanks so much for clarifying the behaviour!