Description
At Etsy, we leverage our internal A/B testing tool when we launch new campaigns, polish the look and feel of our site, or even make changes to our search and recommendation algorithms. As our experimentation platform scales and the velocity of experimentation increases rapidly across the company, we also face a number of challenges. In this talk, I will talk about how we utilize Python to investigate and evaluate one of the problems, “peeking” at results early in order to detect maximum significance with minimum sample size.
We assessed the overall problem and how it affected experiments at Etsy. We then evaluated a few solutions that have been proposed and applied in industry and academia, keeping in mind the unique challenges we face as a fast-paced e-commerce company. After going through the analysis and evaluation, I will discuss the approach we at Etsy took to tackle the peeking problem.