Playing tasks with Django-Celery

Summary

[EuroPython 2011] Mauro Rocco - 22 June 2011 in "Track Tagliatelle "

Description

Celery is an open source task queueing system based on distributed message passing.

I will talk about the tools that Celery offers for task distribution and how to monitor and manage the system using a Django web interface. This talk will also focus on how we use Celery at Jamendo and our real solutions to some common issues you may encounter when developing a back-office based on Celery.

The talk will cover the following topics:

• A brief overview of Celery and the AMPQ protocol AMPQ protocol overview, Celery introduction: Celery, RabbitMQ code examples

• The impact of Celery on the Jamendo work-flow; examples with real tasks. Here I will talk about the Jamendo back-office infrastructure and some of our common tasks. I will discuss the improvements made by introducing a new back-office system based on Celery. I will show some code snippets and go over some real scenarios.

• Overview of the Django Celery admin interface and some Jamendo extensions. Let's talk about the Django-Celery interface that allows one to monitor or schedule tasks directly from the Django admin. I will explain which common additional features are necessary and how to add them.

• Common "gotchas" we encountered while working with Celery and how we solved them.

• Global task locks

• Centralized logging: be able to read all the logs of all celery workers on different servers and filter them for real-time debugging