The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon) | Lenny's Podcast: Product | Growth | Career

This podcast discusses experimental strategies for product development, and addresses common challenges and limitations of A/B experiments. The speakers delve into the importance of defining an Overall Evaluation Criterion and countervailing metrics to avoid the sunk cost fallacy, and underscore the value of experimentation and redesign guided by data-driven decisions.

Outlines

Sign in to continue reading, translating and more.

Continue

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Lenny's Podcast: Product | Growth | Career

The Art of Experimentation: Uncovering Insights Through A-B Testing

Uncovering Revenue Gems: The Art of Small Wins in Experimentation

Experimentation: Learning from Successes and Failures

Experimentation Strategies: Balancing Incremental Changes and High-Risk Ideas

Navigating A-B Testing: Practical Considerations and Strategies

Optimizing for Long-Term Growth: Balancing Revenue and User Experience

The Art of Incremental Changes: Avoiding the Sunk Cost Fallacy in Product Design

Experimentation and Redesign: Balancing Innovation and Risk

Experimentation and Decision-Making at Airbnb: A Former Insider's Perspective

Trustworthy Experimentation: Insights from the Author of 'Trustworthy Online Controlled Experiments'

Unveiling the Pitfalls of Experimentation: Sample Ratio Mismatch and Its Consequences

Navigating the Complexities of Experimentation: Statistical Significance, False Positives, and Replication

Optimizing Experimentation: Building Trust, Speed, and Platforms

Discovering Insights: Books, Products, and Experiences

Structured Narratives, Hierarchy of Evidence, and A-B Testing: Insights from Ronnie Batista

The ultimate guide to A/B testing | Ronny Kohavi (Airbnb, Microsoft, Amazon)

Lenny's Podcast: Product | Growth | Career

00:00The Art of Experimentation: Uncovering Insights Through A-B Testing

The Art of Experimentation: Uncovering Insights Through A-B Testing

06:44Uncovering Revenue Gems: The Art of Small Wins in Experimentation

Uncovering Revenue Gems: The Art of Small Wins in Experimentation

13:03Experimentation: Learning from Successes and Failures

Experimentation: Learning from Successes and Failures

18:54Experimentation Strategies: Balancing Incremental Changes and High-Risk Ideas

Experimentation Strategies: Balancing Incremental Changes and High-Risk Ideas

24:03Navigating A-B Testing: Practical Considerations and Strategies

Navigating A-B Testing: Practical Considerations and Strategies

28:53Optimizing for Long-Term Growth: Balancing Revenue and User Experience

Optimizing for Long-Term Growth: Balancing Revenue and User Experience

33:16The Art of Incremental Changes: Avoiding the Sunk Cost Fallacy in Product Design

The Art of Incremental Changes: Avoiding the Sunk Cost Fallacy in Product Design

39:15Experimentation and Redesign: Balancing Innovation and Risk

Experimentation and Redesign: Balancing Innovation and Risk

44:26Experimentation and Decision-Making at Airbnb: A Former Insider's Perspective

Experimentation and Decision-Making at Airbnb: A Former Insider's Perspective

49:26Trustworthy Experimentation: Insights from the Author of 'Trustworthy Online Controlled Experiments'

Trustworthy Experimentation: Insights from the Author of 'Trustworthy Online Controlled Experiments'

55:34Unveiling the Pitfalls of Experimentation: Sample Ratio Mismatch and Its Consequences

Unveiling the Pitfalls of Experimentation: Sample Ratio Mismatch and Its Consequences

1:00:25Navigating the Complexities of Experimentation: Statistical Significance, False Positives, and Replication

Navigating the Complexities of Experimentation: Statistical Significance, False Positives, and Replication

1:06:20Optimizing Experimentation: Building Trust, Speed, and Platforms

Optimizing Experimentation: Building Trust, Speed, and Platforms

1:12:56Discovering Insights: Books, Products, and Experiences

Discovering Insights: Books, Products, and Experiences

1:18:33Structured Narratives, Hierarchy of Evidence, and A-B Testing: Insights from Ronnie Batista

Structured Narratives, Hierarchy of Evidence, and A-B Testing: Insights from Ronnie Batista