ChannelPulse
Randomisation, sample-ratio mismatch, and guardrails
Before an experiment can be right about its effect it has to be right about its assignment. This lesson picks the unit of randomisation, computes the sample-ratio-mismatch statistic that invalidates a test outright, shows an aggregate check passing while a per-day check fails, and defines the guardrail metrics that stop a winning test from shipping damage.
What you'll be able to do
- Choose a unit of randomisation and say what each choice makes measurable and what it breaks
- Compute the sample-ratio-mismatch chi-square statistic and state what it invalidates
- Detect an assignment bug that an aggregate check misses but a per-day breakdown exposes
- Define guardrail metrics and pre-treatment checks, and say which of them should block a launch
Before this: the-two-errors-and-the-size-of-the-test
The rest of this lesson is in the app
Before an experiment can be right about its effect it has to be right about its assignment. This lesson picks the unit of randomisation, computes the sample-ratio-mismatch statistic that invalidates a test outright, shows an aggregate check passing while a per-day check fails, and defines the guardrail metrics that stop a winning test from shipping damage. This walkthrough runs about 26 minutes, with runnable code you can edit and re-run as you read.
Continue in ChannelPulseThe first module of every track is free to read on the web — see what's open in Data science.