ChannelPulse
Diagnosing a metric drop
"Signups fell 8% — go." The round is lost by candidates who start guessing causes and won by candidates who localise first. This lesson is the order of operations, the contribution arithmetic that finds the responsible segment, and the mix shift that makes a metric fall while every segment improves.
What you'll be able to do
- Run a metric-drop investigation in an order that eliminates whole classes of cause at a time
- Rule out instrumentation and seasonality before proposing a single product hypothesis
- Localise a drop by segment using contribution rather than percentage change
- Detect a mix shift, and explain a fall in the total while every segment improved
Before this: metrics-you-can-defend
"Daily signups are down 8% week over week. What do you do?"
Watch what most candidates do with that sentence: they start generating causes. Maybe a competitor launched. Maybe the funnel broke. Maybe marketing cut spend. Each is plausible, none is checkable from where they're sitting, and twenty minutes later there are nine hypotheses and no evidence. The interviewer's note says unstructured.
The move is to refuse to hypothesise until you have localised. A drop is a number, and numbers can be taken apart. Every part you eliminate deletes a whole family of causes for free.
The same video the first lesson of this module used, but only one chapter of it matters here — 04:17 to 08:01 is the diagnosis framework. Watch that four-minute stretch, then read the order of operations below, which is the same idea taken one level further: each stage eliminates a whole family of causes rather than testing one.
Jump to the part you need
Order of operations
Work outside in. Each stage is cheaper than the one after it and can kill the investigation outright, which is exactly why it goes first.
| # | Ask | Why it comes here |
|---|---|---|
| 1 | Is the metric real? | A logging change, a bad deploy of the tracking SDK, a redefinition in the dashboard. The most common cause of a metric moving is a change to how it's counted, and it costs one question to eliminate |
| 2 | Is it new? | Compare with the same week last year and the surrounding weeks. Weekly seasonality, a holiday, or the tail of a promo that inflated the prior week — a drop from an artificial peak is not a drop |
| 3 | Is it everywhere? | Segment by platform, geography, app version, new vs returning, acquisition channel. This is the step that pays, and the section below is how you do it |
| 4 | Is it one step or the whole funnel? | A drop concentrated at one transition points at that transition. A uniform drop across all steps points at traffic mix, not at the product |
| 5 | Internal or external? | Only now. Our release, our experiment, our pricing — versus a competitor, a platform policy, an app-store ranking change, the news |
Localise with contribution, not percentage change
Stage 3 is where the answer usually lives, and there is a specific arithmetic mistake to avoid. Segments have different sizes, so a segment's own percentage change tells you how badly it is doing while telling you nothing about how much of the total drop it explains. The number you want is its contribution: its absolute change as a share of the total absolute change.
A drop is a weighted sum of segment drops, and usually one term dominates. Split the same "signups down 8%" week by platform:
| Segment | Before | After | Absolute change | Its own change | Share of the total loss |
|---|---|---|---|---|---|
| iOS | 42,000 | 41,580 | −420 | −1.0% | 4.9% |
| Android | 38,000 | 30,020 | −7,980 | −21.0% | 92.8% |
| Web desktop | 15,000 | 14,850 | −150 | −1.0% | 1.7% |
| Web mobile | 5,000 | 4,950 | −50 | −1.0% | 0.6% |
| Total | 100,000 | 91,400 | −8,600 | −8.6% | 100% |
The same four rows as the table, drawn to scale. Android's bar is roughly nineteen times the height of the next one, and that is the entire finding — the other three platforms are each down about 1%, which is noise you would see in a normal week. Percentage change would have shown you four numbers of similar size; contribution shows you one.
That changes the investigation completely. The headline was "down 8.6%," which sounds like something happened to the product. The reality is a 21% drop on Android and roughly a 1% wobble everywhere else — and Android accounts for 92.8% of the loss.
Notice how much that deletes. A competitor launch would not be Android-only. A pricing change would not be Android-only. Marketing spend cuts would not be Android-only unless the campaign was Android-targeted, which is a single question. What survives is a short, testable list: an Android release, a Play Store review or ranking change, an SDK or OS update, a Google sign-in change, or an Android-specific experiment. You have gone from nine guesses to five checkable ones without ever speculating, and the next question is obvious: what shipped to Android that week?
The case where nothing is broken
Now the failure that catches strong candidates, because the data appears to contradict itself.
Conversion fell, and every single segment improved. Both are true at once:
| Segment | Visits before | Rate before | Visits after | Rate after | Share of visits |
|---|---|---|---|---|---|
| Desktop | 60,000 | 10% | 30,000 | 11% | 60% → 30% |
| Mobile | 40,000 | 4% | 70,000 | 5% | 40% → 70% |
| Overall | 100,000 | 7.60% | 100,000 | 6.80% |
Split the change into its two forces by holding one fixed at a time — weights at their old values to isolate the rate effect, rates at their old values to isolate the mix effect, and the cross-term for what is left over:
| Force | Worth |
|---|---|
| Rate effect — segments got better | +1.00 pp |
| Mix effect — traffic moved | −1.80 pp |
| Interaction | 0.00 pp |
| Sum | −0.80 pp — exactly the change that happened |
Desktop conversion went from 10% to 11%. Mobile went from 4% to 5%. Both teams shipped improvements, both can prove it, and the company-wide number fell from 7.60% to 6.80%.
Nothing is wrong with the data. Traffic moved. Desktop went from 60% of visits to 30% while mobile went from 40% to 70%, and mobile converts at less than half the desktop rate. The average fell because the weights changed, not because any rate did.
Pricing the two forces separately is what gives you the sentence to say in the room — not "it's Simpson's paradox," which names the phenomenon without explaining anything, but "the segments each improved by about a point; the total fell because 30% of our traffic moved to a surface that converts at half the rate, and that mix change is worth −1.8 points on its own."
And now the important question, which is not analytical: is this a problem? It depends entirely on why the mix moved.
- If mobile traffic grew because a campaign brought in lower-intent users, the aggregate metric fell and nothing is broken. Report per-segment rates and stop looking.
- If desktop traffic halved, that is the actual story and the conversion metric was a distraction from it. Ask why 30,000 desktop visits disappeared.
- If users are switching from desktop to mobile for the same task and converting worse there, then the mix shift is real and the fix is the mobile experience.
Three different conclusions from identical arithmetic. Which is why the answer to a metrics question is never a number.
What to say, in order
The compressed version, which is also a decent script:
- "First, is the metric real and is it new?" Instrumentation and seasonality, two sentences, then move on.
- "Now let me localise it." Platform, geography, version, new versus returning, channel — and use contribution, not percentage change, to decide which segment matters.
- "Is it the whole funnel or one step?" One step implicates that step; all steps implicate the traffic.
- "Did the mix change?" Check before concluding anything about the product. Every segment can improve while the total falls.
- "Here's the shortlist and how I'd confirm it." Two or three causes that survive the localisation, and the specific query or log that would confirm each.
- "And here's what I'd do about it today." A diagnosis with no action is half an answer. Mitigation before root cause is often correct — roll back first, understand afterwards.
That last point is where this module closes. You now have the method the whole track rests on: you can read the room, open a product answer with a segment you sized rather than sensed, choose a metric that decomposes and an experiment that can actually see, and take a number apart without guessing. What comes next is the machinery each of those leans on — prioritisation that survives being asked "why not the other thing," strategy questions where the answer depends on the specific company, and the technical fluency that decides whether engineers believe you.