Contribute Media
A thank you to everyone who has made this possible: Read More

Building Distributed System with Celery on Docker Swarm

Description

In this talk, the basic mechanisms of Celery and Docker-Swarm will be explained. With Docker-Swarm , a cluster will be built upon two Raspberry Pi machines. Hadoop entry-level "Word Count" program will be re-writen in Python and executed parallelly via Celery on the cluster. An example of distributed system modeling nerual-network will also be explained.

Abstract

In the territory of Python, Celery is a famous distributed task queue framework, its Canvas mechanism is specially powerful in building complex workflow. Celery can function in distributed environment and go perfectly along with Docker-Swarm: Docker-Swarm provides a cluster environment and the worker containers to sustain Celery; worker containers and processes and be dynamically scaled and expanded to fulfill Celery's need, work together parallelly to accomplish the computation. With Celery and Docker-Swarm, an example will be provided, in which a cluster of 8 containers will be built upon two Raspberry Pi machines. Hadoop entry-level "Word Count" program will be re-writen in Python and executed parallelly via Celery on the cluster.

Regarding concurrent/distributed computing, IoT, microservices, serverless-computing are all well known paradigms. Based on MQTT, IBM Bluemix is one of the leaders in this field, however, free of charge only for 30 days. With Celery and Docker-Swarm, and with respect to Bluemix, we can actually build a private IoT-liked system in minutes, free of any charge. To provide an example, a distributed system using Celery and Docker-Swarm modeling a neural-network will be explained.

Improve this page