ChannelPulse
Leakage — how a model gets 99% and helps nobody
Leakage is the failure that looks like success, and it is the most common reason a model that validated well does nothing in production. This builds a dataset with no signal whatsoever and extracts 69% accuracy from it by encoding a feature on the wrong rows, then writes the join that quietly counts events from after the prediction date and the as-of join that fixes it.
What you'll be able to do
- Define leakage as information available at training time and absent at prediction time
- Produce apparent accuracy from a dataset containing no signal, and explain the mechanism
- Write a feature query with an as-of time bound, and recognise the join that leaks
- Choose a validation split that matches how the model will be used, and list the audits that catch leakage before launch
Before this: the-two-errors-and-the-size-of-the-test
The rest of this lesson is in the app
Leakage is the failure that looks like success, and it is the most common reason a model that validated well does nothing in production. This builds a dataset with no signal whatsoever and extracts 69% accuracy from it by encoding a feature on the wrong rows, then writes the join that quietly counts events from after the prediction date and the as-of join that fixes it. This walkthrough runs about 26 minutes, with runnable code you can edit and re-run as you read.
Continue in ChannelPulseThe first module of every track is free to read on the web — see what's open in Data science.