ChannelPulse
Calibration and thresholds — turning a score into a decision
A model outputs a number; a product needs an action. This lesson sweeps a threshold and shows that neither precision nor recall picks it — the cost of the two mistakes does, at exactly COST_FP/(COST_FP+COST_FN) — then builds a reliability table proving that a model with identical ranking can be badly miscalibrated, and that the cost rule silently breaks when it is.
What you'll be able to do
- Derive the cost-optimal decision threshold from the cost of a false positive and a false negative
- Explain why precision, recall, F1, and AUC cannot choose a threshold
- Build a reliability table and read miscalibration off it
- Choose between Platt scaling, isotonic regression, and re-weighting, and say when calibration does not matter
Before this: leakage
The rest of this lesson is in the app
A model outputs a number; a product needs an action. This lesson sweeps a threshold and shows that neither precision nor recall picks it — the cost of the two mistakes does, at exactly COST_FP/(COST_FP+COST_FN) — then builds a reliability table proving that a model with identical ranking can be badly miscalibrated, and that the cost rule silently breaks when it is. This walkthrough runs about 27 minutes, with runnable code you can edit and re-run as you read.
Continue in ChannelPulseThe first module of every track is free to read on the web — see what's open in Data science.