Metrics you can decompose and defend

"How would you measure the success of this?" is the question where competent answers become interchangeable. Almost everyone produces the same shape: a north-star metric, some supporting metrics, a guardrail. The list is correct and it is not worth many points, because the interviewer has heard it four times this week.

What earns the round is the two things nobody does: decomposing the metric until it touches something a team could actually change, and checking whether the change you are proposing is even detectable.

A north star that doesn't decompose is a slogan

A north-star metric earns its name by being the thing you would move if you could only move one. The test of whether you have chosen well is whether it breaks into factors that map onto work.

Take retained users per month as the north star, and write it as the funnel that produces it:

retained users/mo = visitors × signup rate × activation rate × week-4 retention

At 2,000,000 monthly visitors, an 8% signup rate, 35% activation (completing the first real action) and 42% week-4 retention of activated users, that is 23,520 retained users a month.

stagevolumestep

Monthly visitors 2,000,000 —

Signed up 160,000 8.0%

Activated 56,000 35%

Retained at week 4 23,520 42%

volume reaching this stage

Each bar is a factor in the north star, and the right-hand column is that factor's own rate — 8%, 35%, 42%. End to end that is 1.18%, which is the product of the steps and not their average. Resist the obvious read: the biggest step loss is signup, but as the next two tables show, step size does not rank the work and headroom does.

Now apply the same 10% relative improvement to each input in turn:

That identical column is the useful surprise, and it is worth sitting with.

A 10% relative lift in signup rate, activation rate, or retention produces exactly the same +10.0% in the north star. That is not a quirk of these numbers; it is arithmetic. When your metric is a product of factors, it is equally sensitive to a relative change in any of them, so the sensitivity analysis every candidate reaches for cannot rank the work. If you have ever heard someone argue that retention is "higher leverage" than activation, this is the number that says the claim needs a different justification.

What does rank the work is headroom — how far each rate sits from a ceiling you can defend:

Activation at 35% against a plausible 70% has room to double; retention at 42% against 50% has 19%. So activation is where the quarter goes, and the last column prices it: at the ceiling the north star roughly doubles.

Counter-metrics: the term is unstable, so define it out loud

The vocabulary here genuinely is not standardised. Some organisations reserve "guardrail" for cross-product or ecosystem health and use "counter-metric" for harms inside one product; others use "guardrail" for both; practitioners writing about it openly disagree about where the line falls.

The practical consequence is that you should not guess which definition your interviewer holds. Offer yours in one clause and move on:

"I'll use counter-metric for harm inside the feature and ecosystem guardrail for harm to the rest of the product — happy to use whatever split you use."

That sentence costs four seconds and removes an entire category of crossed wires. It also demonstrates something real: that you know these terms are conventions rather than laws.

Underneath the vocabulary the concept is simple and singular — measure what could go wrong — and it splits usefully in two:

The arithmetic that ends the argument

Now the part that most candidates skip and that most reliably impresses: whether the experiment can detect what you claim.

WatchEmma Ding · 16:36

Covers the failure modes a power calculation alone will not save you from. The three worth your time are multiple testing, novelty effects, and interference between groups — each one is a way a correctly-powered test still returns a number you should not act on.

Crack A/B testing problems in interviews Play “Crack A/B testing problems in interviews” on YouTube

The sample size for a two-sample comparison of means is standard, and the NIST/SEMATECH handbook states it plainly: with the standard deviation assumed known, the minimum N for a two-sided test at significance level α and power 1−β to detect a shift δ is

N = (z(1−α/2) + z(1−β))² · (σ/δ)²

Everything that matters about experiment design is in that formula, and in particular the fact that δ is squared. Sanity-check it against the handbook's own worked example first: one-sided, α = 0.05, β = 0.10, detecting a shift of one standard deviation gives (1.645 + 1.282)² × (1/1)² = 8.567, which rounds up to the 9 printed in the handbook.

A conversion rate is a proportion rather than a mean, so σ² = p(1−p), and a two-arm test needs that variance twice:

N per arm = (z(1−α/2) + z(1−β))² · 2p̄(1−p̄) / δ²

where p̄ is the pooled rate under the alternative. At a 10% baseline checkout conversion, 90% power and α = 0.05, two-sided:

That is the table that should change how you propose experiments. Because the effect size is squared, halving the effect you want to detect quadruples the sample you need. Detecting a 20% relative lift on a 10% baseline takes about 5,100 users per arm. Detecting a 1% lift takes about 1.9 million — 370 times as many — and at 25,000 users per arm per day that is a 76-day experiment.

This is the arithmetic that resolves the most common bad meeting in product management. Someone proposes a small change, claims a 1% conversion lift, and asks to A/B test it for a week. The test is not underpowered by a little; it has almost no chance of detecting the claimed effect, and it will come back "no significant difference" regardless of whether the change worked. Two weeks are spent, and the organisation learns nothing while believing it ran an experiment.

What a strong answer sounds like

Compressed to the moves, since the shape matters more than the words:

1. One north star, decomposed. Name it, then break it into factors, then say which factor this work touches. A metric that doesn't decompose isn't a metric, it's a mission statement.
2. Rank by headroom, and name the ceiling's source. Sensitivity can't rank a product of factors; distance-from-ceiling can.
3. A counter-metric that could fire. Name the mechanism by which your change could do harm, then the number that would show it.
4. Power before launch. "At our traffic this needs eleven days for the effect I'm claiming; if we only have a week, I'd rather test a bigger change than a better-instrumented small one."
5. Say what would make you kill it. The threshold, stated in advance, before you have seen data you are motivated to reinterpret.

That last one is the habit the next lesson depends on. Deciding what a number means after you have seen it is how a metric drop turns into three weeks of confident, unfalsifiable theories — which is exactly the failure mode we take apart next.