ChannelPulse
Reading an experiment result
PMs are rarely asked to design a test and are constantly asked to act on one. This lesson works through the four results you actually get — flat, mixed, significant but tiny, and significant with a guardrail regression — shows what a flat result does and does not rule out at a given sample size, and ends with the ship decision when the evidence is genuinely ambiguous.
What you'll be able to do
- Say what a flat result rules out at a given sample size, and what it leaves open
- Decide on a mixed result without cherry-picking the segment that agrees with you
- Separate statistical significance from practical significance, and price the change either way
- Make a ship / iterate / kill call, with the reasoning that makes it defensible afterwards
Before this: metrics-you-can-defend
The rest of this lesson is in the app
PMs are rarely asked to design a test and are constantly asked to act on one. This lesson works through the four results you actually get — flat, mixed, significant but tiny, and significant with a guardrail regression — shows what a flat result does and does not rule out at a given sample size, and ends with the ship decision when the evidence is genuinely ambiguous. This walkthrough runs about 25 minutes, with runnable code you can edit and re-run as you read.
Continue in ChannelPulseThe first module of every track is free to read on the web — see what's open in Product management.