9,008 questions
- Bountied 0
- Unanswered
- Frequent
- Score
- Trending
- Week
- Month
- Unanswered (my tags)
0
votes
0
answers
32
views
Why does Celery with Redis backend create hundreds of connections despite max_connections limits
I am using Django + Celery with Redis as both the broker and the result backend. Here is my Celery configuration:
# Celery settings
CELERY_BROKER_URL = 'redis://redis:6379/1'
CELERY_RESULT_BACKEND = '...
0
votes
1
answer
166
views
Celery crashes when PgBouncer closes idle connections (idle timeouts enabled) [duplicate]
I’m encountering an issue when running Celery with PgBouncer and PostgreSQL after enabling idle connection timeouts.
My stack includes:
Django (served via Tornado)
Celery (workers + beat)
...
1
vote
1
answer
63
views
Can I use Celery for a task which calls an external library that uses multiprocessing?
I am hoping to use Celery to manage the task queue for my application, and am wondering if it is capable of managing tasks that themselves use multiprocessing, called from an external library. For ...
0
votes
0
answers
57
views
How to avoid celery_delayed_0-27 queues creation?
I use celery 5.5.0 with RabbitMQ 4.1.1 as a message broker. After a switch from classic queues to quorum queues I got quite a lot of queues called celery_delayed_0-27. In total 28 new queues. As far ...
2
votes
0
answers
117
views
How to disable Celery startup logs?
I'm getting a bunch of logs like this:
[2025年11月29日 16:13:15,731]
def group(self, tasks, result, group_id, partial_args, add_to_parent=0):
return 1
[2025年11月29日 16:13:15,732]
def xmap(task, it):
...
1
vote
0
answers
58
views
Django Celery Beat SQS slow scheduling
Beat seems to be sending the messages into SQS very slowly, about 100/minute.
Every Sunday I have a sendout to about 16k users, and they're all booked for 6.30pm. Beat starts picking it up at the ...
1
vote
1
answer
49
views
Chain and on_success handler execution order
I have a Celery Chain
chain(
first_task.s(),
second_task.s(),
third_task.s(),
).apply_async()
The tasks implement the on_success and on_failure handler functions to send messages to ...
Arttu's user avatar
- 1,013
0
votes
1
answer
78
views
How to keep celery tasks in redis after restart
I am using redis and celery in docker-compose. I noticed that I am loosing celery tasks when I am restarting docker-compose (docker-compose down, and then docker-compose up)
How to improve my redis ...
0
votes
1
answer
53
views
I can only run my backend tests locally because all the instances of the mocked environment are created and into the actual db because of celery
I want to create tests but every time I run a test it triggers celery and celery creates instances into my local db. that means that if I run those tests in the prod or dev servers, then it will ...
4
votes
2
answers
223
views
Celery memory leak in Django — worker memory keeps increasing and not released after tasks complete
I’m using Django + Celery for data crawling tasks, but the memory usage of the Celery worker keeps increasing over time and never goes down after each task is completed.
I’m using:
celery==5.5.3
...
0
votes
2
answers
143
views
asyncio.run with SQLAlchemy asyncio sessions works only once, then raises "Future attached to a different loop" and "another operation is in progress"
I’m running into an issue when using asyncio.run together with SQLAlchemy (async) inside a Celery task.
When I call the function the first time, it works fine.
On the second call, I get:
RuntimeError: ...
2
votes
1
answer
70
views
Celery: How can I kill a task at a specific time?
I want to be sure that a task is killed at a certain time if it is still running. The context is an overloaded worker, where tasks are not picked up straight away.
Image this "busy" worker (...
3
votes
1
answer
79
views
Celery task called inside another task always goes to default queue even with queue specified
I’m running Celery with Django and Celery Beat.
Celery Beat triggers an outer task every 30 minutes, and inside that task I enqueue another task per item.
Both tasks are decorated to use the same ...
0
votes
1
answer
82
views
Why does adding Nmap port scanning inside a Celery task slow down my FastAPI + SQLAlchemy + PostgreSQL app?
I am building a scanning service using FastAPI + Celery + PostgreSQL + SQLAlchemy + Nmap.
I have a Celery task that performs WHOIS, DNS lookups, IP lookups, and then port scanning using python-nmap. ...
1
vote
1
answer
120
views
How to remove intermediate results after execution of the chord without blocking execution?
Here’s the pattern I want:
Dispatch multiple tasks in parallel.
Aggregate all their results into a final result.
Remove the intermediate results right after the chord result is ready, without ...