How to activate the process queue in "django-background-tasks"

Question

I am new to Django and the django-background-tasks package.

I am facing an issue that I couldn't do/start background task unless I forcefully run the command process_tasks , that is python manage.py process_tasks. I want to do/start background task without run the process_tasks command.

settings.py

MAX_ATTEMPTS=1
BACKGROUND_TASK_RUN_ASYNC = True

tasks.py

from background_task import background
#included necessary packages for SMTP
@background(schedule=5)
def test():
 #send mail to some ids

views.py

def index(request):
 test(schedule=5)
 return HttpResponse("Hello, world. ")

Ignore my logic.

Answer 1

In case anybody else is trying to get this working - I call a shell script from my Dockerfile that then runs two commands. One to start the webserver (gunicorn or run_server) and one to run the manage.py process_tasks process. The important thing is to run the process_tasks process as a background task.

At the bottom of my Dockerfile I have: CMD ["./run_app.sh"]

And my run_app.sh file looks like:

#!/usr/bin/env bash
# start background tasks 
python manage.py process_tasks &
gunicorn --workers=3 app.wsgi -b 0.0.0.0:8080

Note the trailing & from the process tasks command. This enables the shell script to continue processing and run the gunicorn command.

Hope this helps someone.

Answer 2

Well you could make a bash script that makes the process_tasks call and call that script from python

to call the script from python you can use

import subprocess
subprocess.call("process_tasks.sh", shell=True)

as its explained here Running bash script from within python

your bash script may look something like this

cd your/virtualenv/directory
source virtualenv/bin/activate
exec /path/to/manage.py process_tasks

dont forget to make process_tasks.sh executable

Answer 3

You must run process_tasks command to execute tasks that have been scheduled. There is no other way. Process_tasks checks the db regularly for scheduled tasks and executes them in the background.

In a local environment

Open a terminal, cd into your app folder and launch the command python manage.py process_tasks. If you are using a virtualenv make sure you activate it first.

In a production environment

Option 1: Run a cron job

One solution is to launch the process_tasks command using a cron job. Beware that cron launches a new process at the scheduled interval without caring about the previous processes. You must make sure the process ends by the time the new cron call is scheduled. Otherwise you will be wasting resources on parallel instances of process_tasks. Happily this check is fairly easy to do.

To launch process_tasks for a limited time you can use its duration parameter. For example, to launch it with a life-span of 10 hours you call:

python manage.py process_tasks --duration 36000 

Configure the cron job: In cPanel you can use the Cron Jobs app. Configure the cron job to run hourly and add the following lines as your cron command:

cd /home/username/appname && /home/username/virtualenv/bin/python /home/username/appname/manage.py process_tasks --duration 3540 

This will cd into your app folder and launch the process_tasks command, taking into consideration your virtual environment and using a life-span of 59 minutes for the process_tasks command.

Update the paths to you app & virtual environment accordingly! Adjust the cron job interval and duration parameter to fit your needs!

Benefits:

• If the command fails it will get restarted by the cron job later.
• You don't have to worry about resource/memory leaks too much.

Option 2. Launch command at server startup

Basically you will have to create a script file containing the same launch command and configure your server to run it at startup.

You lose the benefit that your process is restarted from time to time so you will probably have to monitor its health state. If you are going down this path, a good approach is to use supervisord to monitor your process.

Answer 4

That is how django-background-tasks works. You have to run that process for the task runner to start processing tasks.

https://github.com/arteria/django-background-tasks#running-tasks

• user7139313: Can I do background task without executing python manage.py process_tasks command ? ? Or can I automate that process ?
• chadgh: From what I can tell in the documentation you have to run the command. I use this in production at my work. We run our Django application process (uwsgi or gunicorn or something like that) and then we run the process_tasks command that churns through the tasks.
• Jed: Can you explain more about how you're activating the background-tasks process queue as per the op's question. I've got the same question. The only clear answers I've found are similar to Faiya's above, either running via a separate bash script or executing the manage.py command directly and leaving it live via tmux or something similar. I would like to know, however, if there is a more direct way to call the manage.py command via the wsgi.py file or somewhere else within the django stack. It sounds like you've got experience and any suggestions would be appreciated.
• chadgh: I've done this in two ways. uwsgi can run cron jobs for you. I like the cron2 syntax in the configs - uwsgi-docs.readthedocs.io/en/latest/… . An example config for this might be cron2 = minute=-1 python path/to/manage.py process_tasks -d 30 This will run the task for 30 seconds every minute. Another way I've done this is using Supervisor. supervisord.org

Answer 5

You can call the manage.py command from directly in your django project using the below code:

from django.core.management import call_command
call_command('process_tasks')

check out the docs here

• zoran: Doesn't work in this case. You will get django.core.management.base.CommandError: Unknown command

Answer 6

I am also trying to use Django Background Tasks and have done successfully everything and my task execute perfectly when run manually "python manage.py process_tasks" but my background task does not trigger as per @background(schedule=60) automatically without running "python manage.py process_tasks" or whatever I set schedule time. than what is the point of setting schedule time for a task. When task has to be executed with command python manage.py process_task is mandatory to trigger all background task.

• Bruce: Did yo ever find a away to run it automatically

Answer 7

python manage.py process_tasks &

gunicorn --workers=3 app.wsgi -b 0.0.0.0:8080