CUPED Explained: Why you must know it in AB testing?

This podcast episode features an interview with Kenneth, a statistician from Meta with a PhD in mathematics from UC Berkeley, who specializes in experimentation. The discussion centers around CUPED (Controlled-experiment Using Pre-Experiment Data), a technique also known as regression adjustments, used to reduce variance in A/B testing without introducing bias. Kenneth explains the history, theoretical underpinnings, and practical applications of CUPED, including how it can lead to quicker statistical results and more efficient experimentation. He uses an example of testing a height-growth pill to illustrate how CUPED leverages pre-experiment data to adjust for statistical noise. The conversation covers the importance of ensuring that input variables are not impacted by the experiment, the use of regression analysis to predict outcomes, and considerations for applying CUPED to ratios like conversion rates. Kenneth also touches on the limitations of variance reduction and the trade-offs between complex modeling and trustworthiness in experimentation.