Using Message Queues for Distributed Processing

Summary

Message queues provide a reliable, language agnostic method for sharing data between processes for distributed processing, and to manage work flows. This talk will examine the role of message queues in a redesign of the provisioning process for Texas A[HTML REMOVED] University's enterprise directory.

Description

About six months ago we moved our account provisioning process from a monolithic Perl script to a set of distributed, interconnected python scripts. By using message queues (specifically beanstalkd) to pass messages between the processes, we were able to build a system that was much faster, more reliable and more flexible.

I will walk through the process that led us to choose beanstalkd, look at some alternatives that are available, and describe some use cases for message queues.