Asynchronous Task Queues

Background

What

Work performed in another process/thread
Work performed on another machine or a pool of machines

Why

Increase User-perceived performance
Avoid IO limitations (slow, unreliable, etc)
Scale horizontally
Avoid GIL in some languages (Ruby, Python)

When

Work considered optional at the time of execution
Web development: action(s) not required to build HTTP response

Basic Concept

Client pushes message onto queue
Worker shifts message off queue
Worker executes code based on message

Architecture

Message Serialization

Structured payload format
Cross-platform type handling
Speed / Payload size

Examples

JSON
msgpack
YAML

Abstractions

Generic interface to many job processors
Allow libraries to perform async work

Examples

Active Job
???

Strategies

Pub/Sub

Most message brokers (& PG with LISTEN/NOTIFY)
Less chance of race conditions - one message per worker
CPU efficient - block until recv

Polling

Normal with Database message bus
Race condition prone

Job processors

Batteries included
Consistent interface
Job retry / worker durability
Some have concurrency built in

Language	Message Broker
Celery	Python	Redis, Beanstalk, MongoDB, CouchDB, Database
Sidekiq	Ruby	Redis
Resque	Ruby	Redis
Delayed Job	Ruby	Database

Message Brokers

Accept messages from clients
Determine possible worker via queue names
Notify worker of job

Examples

Beanstalk
RabbitMQ
Redis
Database
IronMQ
Amazon SQS
many, many more

Best Practices

Use small stateless messages
Keep data out of queue
Queue should have actions
Use a simple serialized type (JSON?)
Idempotent and transactional actions

Demo

Thank You

Github: github.com/adam12
Twitter: @adamrdaniels
Sourcecode: github.com/adam12/asynchronous-task-queues-devtricks-2014

← →

/