Description
Axel Arnold
Having started as a computational physicist, developing GPU-accelerated based molecular dynamics software for 14 years, I developed a passion for Python as scripting frontend and powerful numeric analysis tool. And NumPy makes teaching numerical math to 3rd year physics students so much easier! Eventually, I decided to make my living using Python and joined solute GmbH as Python developer. I still have a passion for high throughput systems, but now not only from developer's, but also the operations point of view.
Abstract
billiger.de is one of the largest price comparison websites in Germany. In this talk, we want to share how we built the scalable, event-driven processing system which renders the products for our website using Python, Elasticsearch and redis.
Description
billiger.de is one of the largest price comparison websites in Germany. Tens of millions of offers need to be clustered into products, with a considerable amount of offers changing every few hours. In this talk, we want to share our experiences and the challenges we were facing on our way to building a scalable, event-driven processing system which renders the products you will eventually find on our website. We give an overview of that system, which consists of a number of services written in Python, a Redis based queueing system and an Elasticsearch cluster for storage. We tell the story of how all of this works together and how it allows us to scale storage and processing throughput according to our needs.
Recorded at PyCon.DE 2017 Karlsruhe: https://de.pycon.org/
Video editing: Sebastian Neubauer & Andrei Dan
Tools: Blender, Avidemux & Sonic Pi