ChannelPulse
Peeking, novelty, and interference — four ways a valid test lies
The arithmetic can be right and the decision still wrong. This simulates 2,000 A/A tests to show peeking turning a 5% false-positive rate into 20%, then covers the novelty effect that fades, the interference that puts your treatment inside your control, and the multiple comparisons that manufacture a winner out of nothing.
What you'll be able to do
- Demonstrate the false-positive inflation caused by repeatedly checking a running experiment
- Choose between a fixed horizon, a sequential test, and a Bayesian stopping rule, and say what each costs
- Recognise novelty and primacy effects, and design the test that distinguishes them from a real effect
- Identify interference and multiple-comparison problems, and correct for both
Before this: randomisation-and-guardrails
The rest of this lesson is in the app
The arithmetic can be right and the decision still wrong. This simulates 2,000 A/A tests to show peeking turning a 5% false-positive rate into 20%, then covers the novelty effect that fades, the interference that puts your treatment inside your control, and the multiple comparisons that manufacture a winner out of nothing. This walkthrough runs about 27 minutes, with runnable code you can edit and re-run as you read.
Continue in ChannelPulseThe first module of every track is free to read on the web — see what's open in Data science.