Description
Manging data at a company of any size can be a pain. Data pipelines and other automation workflows can help! In this talk, we'll cover how to determine if a task or data process is a good candidate for automation and review some likely frameworks you can use to do so: like Spotify's Luigi and Airbnb's Airflow. The talk will also cover potential issues in automation and how to best validate proper data processing with your new pipeline.