ChannelPulse
Data science · The method — the round you are actually in

The two errors, and the size of the test

A p-value answers a question almost nobody asks it, and a sample size follows from four numbers you can state out loud. This lesson pins down what each error rate means, reproduces the NIST handbook's worked sample-size example to prove the arithmetic, and then does the iteration everyone skips.

21 min read Free to read Patterns: power-and-mde, sample-size-from-four-numbers

What you'll be able to do

Before this: which-interview-are-you-in

Two questions get asked in almost every statistics round, and they are the same question wearing different clothes: what does a p-value mean, and how many users do we need. Both have exact answers. The first is a definition people paraphrase wrong; the second is arithmetic people don't attempt.

WatchEmma Ding · 13:10

The useful trick here is that each concept is explained twice — once for a technical audience and once for a non-technical one. That pairing is the actual interview skill: you will be asked to define power to a fellow data scientist and then to a PM in the same loop, and the second is harder.

Jump to the part you need

What the p-value is a probability of

A p-value is the probability of observing data at least as extreme as yours assuming the null hypothesis is true. Everything about it is conditional on that assumption.

Which means it is not:

The claimWhy it's wrong
"There's a 3% chance the null is true"The p-value is computed given the null. It cannot also be evidence about the null's probability without a prior.
"There's a 97% chance the effect is real"Same error, restated. And "real" is not the same as "the size we measured".
"p = 0.049 and p = 0.051 are different findings"They are the same finding on either side of a threshold someone chose by convention.
"p > 0.05, so there is no effect"Failing to reject is not accepting. It is compatible with no effect, and with an effect your test was too small to see.

That last row is the one that separates candidates, because it is where the two error rates live.

The asymmetry matters: α is a property of the test alone, while β only means something relative to an effect size. "What's your false-negative rate?" is not answerable. "What's your false-negative rate for a 2% lift?" is.

WatchEmma Ding · 07:37

Seven minutes on exactly the calculation this section derives, taking each of the four inputs in turn. Watch it first if the formula below looks arbitrary — the chapters at 03:20 through 05:36 are α, β, variance and δ one at a time, which is the order that makes the formula stop being something to memorise.

Jump to the part you need

Two truths, two conclusions, four outcomes
What is true, and what you concluded
No real effect
Correct silence, 1 − α
Type I — false alarm, α
A real effect
Type II — missed it, β
Correct detection, power = 1 − β
the question being decomposed

Read the two branches separately, because that is the mistake the names invite. α lives entirely on the left branch — it is a rate among tests where nothing was happening — and β lives entirely on the right. Neither is a probability that you are wrong, and no experiment ever tells you which branch you are on. Choosing α and β is choosing how much of each error you will accept before you look.

Four numbers give you N

The sample size for a test that must detect a shift δ follows from α, β, δ and the standard deviation σ. The NIST/SEMATECH handbook states it directly:

  • two-sided: N = (z₁₋α/₂ + z₁₋β)² · (σ/δ)²
  • one-sided: N = (z₁₋α + z₁₋β)² · (σ/δ)²

The handbook works one example: a one-sided test at α = 0.05 and β = 0.10, detecting a shift of one standard deviation, needs N = 8.567 ≈ 9. That published number is a fixed point we can check our arithmetic against.

JavaScript
// Sample size for a test that must detect a shift of delta. The NIST/SEMATECH
// Engineering Statistics Handbook (7.2.2.2) gives the closed forms:
//
//   two-sided:  N = (z[1-a/2] + z[1-b])^2 * (sigma/delta)^2
//   one-sided:  N = (z[1-a]   + z[1-b])^2 * (sigma/delta)^2
//
// The handbook's worked example is a ONE-sided test at a=0.05, b=0.10 detecting
// a shift of one standard deviation, and it prints N = 8.567 ~ 9. We reproduce
// that number, then do the part everyone skips.

// Inverse normal CDF (Acklam's rational approximation, ~1e-9 absolute error --
// tighter than the 4 significant figures the handbook quotes, which is why our
// 8.564 differs from its 8.567 in the third decimal).
function z(p) {
  const a = [-3.969683028665376e1, 2.209460984245205e2, -2.759285104469687e2,
             1.383577518672690e2, -3.066479806614716e1, 2.506628277459239];
  const b = [-5.447609879822406e1, 1.615858368580409e2, -1.556989798598866e2,
             6.680131188771972e1, -1.328068155288572e1];
  const c = [-7.784894002430293e-3, -3.223964580411365e-1, -2.400758277161838,
             -2.549732539343734, 4.374664141464968, 2.938163982698783];
  const d = [7.784695709041462e-3, 3.224671290700398e-1, 2.445134137142996,
             3.754408661907416];
  const pl = 0.02425;
  let q, r;
  if (p < pl) {
    q = Math.sqrt(-2 * Math.log(p));
    return (((((c[0]*q+c[1])*q+c[2])*q+c[3])*q+c[4])*q+c[5]) /
           ((((d[0]*q+d[1])*q+d[2])*q+d[3])*q+1);
  }
  if (p > 1 - pl) return -z(1 - p);
  q = p - 0.5; r = q * q;
  return (((((a[0]*r+a[1])*r+a[2])*r+a[3])*r+a[4])*r+a[5]) * q /
         (((((b[0]*r+b[1])*r+b[2])*r+b[3])*r+b[4])*r+1);
}

const ALPHA = 0.05;   // risk of calling a null effect real   (type I)
const BETA  = 0.10;   // risk of missing a real effect        (type II) -> 90% power
const SHIFT = 1.0;    // delta, in units of sigma

const nOneSided = (z(1 - ALPHA)     + z(1 - BETA)) ** 2 / SHIFT ** 2;
const nTwoSided = (z(1 - ALPHA / 2) + z(1 - BETA)) ** 2 / SHIFT ** 2;

console.log("critical values");
console.log("  z[1-a]   =", z(1 - ALPHA).toFixed(3), " (one-sided)");
console.log("  z[1-a/2] =", z(1 - ALPHA / 2).toFixed(3), " (two-sided)");
console.log("  z[1-b]   =", z(1 - BETA).toFixed(3));
console.log("");
console.log("N, sigma known, shift = 1 sigma");
console.log("  one-sided:", nOneSided.toFixed(3), "->", Math.ceil(nOneSided), " (handbook: 8.567 ~ 9)");
console.log("  two-sided:", nTwoSided.toFixed(3), "->", Math.ceil(nTwoSided));
console.log("  the price of not committing to a direction:",
            Math.ceil(nTwoSided) - Math.ceil(nOneSided), "more observations");
console.log("");

// ── the part everyone skips ────────────────────────────────────────────────
// Both formulas above assume sigma is KNOWN. It never is: you estimate it from
// data, which widens the interval, which raises N. Then the t critical values
// depend on degrees of freedom, which depend on N -- the thing you are solving
// for. So you iterate. The handbook does exactly one round and says one is
// usually enough.
//
// Two-tailed Student-t quantile by bisection on the CDF, which we get from the
// incomplete beta function.
function betacf(a, b, x) {
  let qab = a + b, qap = a + 1, qam = a - 1, c = 1, d = 1 - qab * x / qap;
  if (Math.abs(d) < 1e-30) d = 1e-30;
  d = 1 / d;
  let h = d;
  for (let m = 1; m <= 200; m++) {
    const m2 = 2 * m;
    let aa = m * (b - m) * x / ((qam + m2) * (a + m2));
    d = 1 + aa * d; if (Math.abs(d) < 1e-30) d = 1e-30;
    c = 1 + aa / c; if (Math.abs(c) < 1e-30) c = 1e-30;
    d = 1 / d; h *= d * c;
    aa = -(a + m) * (qab + m) * x / ((a + m2) * (qap + m2));
    d = 1 + aa * d; if (Math.abs(d) < 1e-30) d = 1e-30;
    c = 1 + aa / c; if (Math.abs(c) < 1e-30) c = 1e-30;
    d = 1 / d;
    const del = d * c; h *= del;
    if (Math.abs(del - 1) < 3e-16) break;
  }
  return h;
}
function lgamma(x) {
  const g = [76.18009172947146, -86.50532032941677, 24.01409824083091,
             -1.231739572450155, 0.1208650973866179e-2, -0.5395239384953e-5];
  let y = x, tmp = x + 5.5;
  tmp -= (x + 0.5) * Math.log(tmp);
  let ser = 1.000000000190015;
  for (let j = 0; j < 6; j++) ser += g[j] / ++y;
  return -tmp + Math.log(2.5066282746310005 * ser / x);
}
function ibeta(a, b, x) {
  if (x <= 0) return 0;
  if (x >= 1) return 1;
  const front = Math.exp(lgamma(a + b) - lgamma(a) - lgamma(b) +
                         a * Math.log(x) + b * Math.log(1 - x));
  return x < (a + 1) / (a + b + 2)
    ? front * betacf(a, b, x) / a
    : 1 - front * betacf(b, a, 1 - x) / b;
}
/** t quantile: the value t with P(T <= t) = p, for df degrees of freedom. */
function tinv(p, df) {
  const cdf = (t) => {
    const x = df / (df + t * t);
    const tail = 0.5 * ibeta(df / 2, 0.5, x);
    return t > 0 ? 1 - tail : tail;
  };
  let lo = -100, hi = 100;
  for (let i = 0; i < 200; i++) {
    const mid = (lo + hi) / 2;
    if (cdf(mid) < p) lo = mid; else hi = mid;
  }
  return (lo + hi) / 2;
}

console.log("N, sigma ESTIMATED from the data (iterate on df = N - 1)");
let n = Math.ceil(nOneSided);
console.log("  start from the z answer: N =", n);
for (let i = 1; i <= 3; i++) {
  const df = n - 1;
  const t1 = tinv(1 - ALPHA, df), t2 = tinv(1 - BETA, df);
  const raw = (t1 + t2) ** 2 / SHIFT ** 2;
  const next = Math.ceil(raw);
  console.log(
    "  iter " + i + ": df =", String(df).padStart(2),
    " t[1-a] =", t1.toFixed(3),
    " t[1-b] =", t2.toFixed(3),
    " N =", raw.toFixed(1), "->", next,
    next === n ? "  (converged)" : ""
  );
  if (next === n) break;
  n = next;
}
console.log("");
console.log("z said", Math.ceil(nOneSided) + ", t says", n + ".",
            "The handbook's first iteration is 10.6 ~ 11.");
console.log("Not knowing sigma cost", n - Math.ceil(nOneSided),
            "observations --", ((n / Math.ceil(nOneSided) - 1) * 100).toFixed(0) + "% more,");
console.log("on a test that was already tiny. At n = 9 the correction is large");
console.log("because df is small; at n = 900 it vanishes. This is why the z");
console.log("formula is fine for A/B tests and wrong for lab-scale samples.");

Three things in that output are worth saying out loud in a room.

The z critical values reproduce the handbook exactly — 1.645 and 1.282, summing to 2.927, whose square is 8.567. Our 8.564 comes from carrying more digits than the handbook's four significant figures, which is a difference in rounding and not in method. Being able to say why your number differs in the third decimal is a better signal than matching it by luck.

Two-sided costs you two observations here and about 23% in general — the ratio of the squared critical-value sums, (1.960 + 1.282)² / (1.645 + 1.282)², which is 1.23 for any δ and σ. The one-sided test spends its whole α budget in one tail, so it needs less evidence. That is a legitimate saving, and it has a price: you have committed to a direction before seeing data, and a real effect the other way is now invisible to you. Say that trade-off out loud rather than picking one silently.

The t-iteration is the part almost nobody does. Both closed forms assume σ is known. It never is — you estimate it, so the critical values come from a t distribution whose degrees of freedom depend on N, the thing you're solving for. So you iterate: start from the z answer of 9, recompute with df = 8, get 10.6 ≈ 11, recompute with df = 10, get 10.1 ≈ 11, done. The handbook does one iteration and notes that one is usually enough.

Turning that into the answer you actually give

Interviewers rarely ask for N directly. They ask "how long should we run this?" — which is the same question with two extra steps.

  1. State the four inputs, as a decision rather than a default. α = 0.05 because that is the convention and nobody is arguing. β = 0.20 or 0.10 — say which and why. δ is the smallest effect that would change what you do, not the effect you hope for. σ comes from historical data on the same metric.
  2. Compute per-arm N, then multiply by the number of arms.
  3. Divide by the traffic that is eligible, not total traffic. Eligible means: reaches the surface, is in the right locale, and is not excluded by another running test.
  4. Round up to whole business weeks. Behaviour is weekly-periodic; a nine-day test compares one-and-a-bit weekends against one.

The two ways a correctly sized test still lies

Peeking. If you check significance daily and stop when p < 0.05, your real false-positive rate is far above 5% — you have taken many looks and kept the one that crossed. The α you chose applies to a single test at a fixed sample size. Fixes: commit to the horizon in advance, or use a method designed for continuous monitoring (sequential tests, always-valid confidence intervals). Naming the problem is most of the credit; naming a fix is the rest.

Twenty metrics. Testing twenty independent metrics at α = 0.05 gives roughly a 64% chance of at least one false positive — 1 − 0.95²⁰. This is why experiment platforms distinguish one primary metric from guardrails and secondaries, and why multiple-comparison corrections exist. Bonferroni (divide α by the number of tests) is conservative and easy to state; Benjamini–Hochberg controls the false discovery rate instead and is what most platforms use for large metric sets.

What's next

Sizing tells you whether a measurement could have detected what you care about. The next lesson is the same skepticism pointed at models: a classifier can score 0.91 ROC-AUC and be unusable, and the arithmetic that shows why is short enough to run.