A/B Testing and Experimentation
Design, run and read experiments that change product decisions.
Taught by Niraj Thapa · Analytics Lead
What you'll learn
- Explain why randomised experiments establish causation where observational data cannot
- Write a falsifiable hypothesis with one primary metric, secondary metrics and guardrails, and pre-register it
- Calculate sample size and run length from baseline, minimum detectable effect, alpha and power in Python
- Assign users deterministically, detect sample ratio mismatch and avoid the peeking problem
- Analyse conversion and revenue metrics with z-tests, Welch's t-test, bootstrap and confidence intervals
- Apply Bayesian analysis and CUPED, then report results as ship, iterate or stop while avoiding classic pitfalls
About this course
A/B testing is how product, marketing and growth teams find out what actually works instead of arguing about it. This intermediate course teaches you to design, run and read online experiments that stand up to scrutiny. You will write testable hypotheses, choose primary and guardrail metrics, calculate sample size and minimum detectable effect, and understand why peeking at results inflates false positives. You will analyse results with p-values, confidence intervals and effect sizes in Python, check for sample ratio mismatch, and try Bayesian analysis and variance reduction with CUPED. The final lesson covers reporting results to stakeholders and the pitfalls, from novelty effects to multiple comparisons, that quietly invalidate tests. It suits product analysts, growth marketers, data analysts and product managers who already know basic statistics such as means, proportions and the normal distribution. You need working Python with pandas, SciPy and statsmodels. You finish able to run an experimentation programme your leadership can trust.
Course content
7 lessons- 1Why Experiments Beat OpinionsCorrelation versus causation, what randomisation buys you, and the anatomy of an online A/B test.Preview
- 2Hypotheses, Metrics and GuardrailsWrite a hypothesis you can falsify, pick one primary metric, and set guardrails that stop a win from hiding a loss.
- 3Sample Size, Power and Minimum Detectable EffectAlpha, power and MDE, why they trade off, and how to compute the sample size in Python before launch.
- 4Running It Cleanly: Randomisation, SRM and PeekingDeterministic assignment, checking for sample ratio mismatch, and why looking at results early inflates false positives.
- 5Reading Results: p-values, Intervals and Effect SizesAnalyse a finished test in Python, report the lift with a confidence interval, and separate statistical from practical significance.
- 6Bayesian A/B Testing and Variance ReductionProbability that the variant is better via Beta-Binomial simulation, and CUPED for sharper results from the same users.
- 7Reporting Results and Avoiding the Classic PitfallsA results write-up stakeholders can act on, the ship/iterate/stop framework, and the mistakes that quietly invalidate tests.