ChannelPulse
Data science · The method — the round you are actually in

Evaluation that survives a rare positive class

ROC-AUC cannot see class prevalence, which is why a fraud model can score 0.91 and drown its users in false alarms. This lesson holds one model fixed, changes only how many negatives exist, and watches ROC-AUC refuse to move while precision collapses — then turns the threshold into a cost decision.

20 min read Free to read Patterns: prevalence-invariance, threshold-as-a-cost-decision

What you'll be able to do

Before this: the-two-errors-and-the-size-of-the-test

"Our fraud model has 0.91 AUC" is a sentence that can be simultaneously true and worthless, and the interview question that exposes it is short: what's the base rate?

The reason is one line of algebra. Look at what each axis divides by.

  • TPR = TP / (TP + FN) — divided by all actual positives.
  • FPR = FP / (FP + TN) — divided by all actual negatives.
  • Precision = TP / (TP + FP) — divided by everything you flagged.

TPR and FPR are within-class rates. Each one is computed inside a single class and normalised by that class's fixed size, so neither can see the ratio between the classes. Precision spans both: its denominator mixes true and false positives, so it feels every extra false alarm the negative pool produces.

WatchHello Interview · 09:44

Recommendation is the most extreme imbalance problem in production — millions of candidates, a handful of positives — and this is how the industry deals with it: never evaluate a single model against the whole catalogue. Watch the naive-approaches chapter for what breaks, then the multi-stage chapter for the fix, because "which stage is this metric measuring?" is the question that saves you in this round.

Jump to the part you need

Watch it happen

Fix one model by fixing its two score distributions. Change nothing but the number of negatives.

JavaScript
// The claim to test: ROC-AUC is invariant to class prevalence and PR-AUC is not.
// If that is true, a model can look excellent and be useless, and you can watch
// it happen without changing the model at all.
//
// So: fix ONE model by fixing its two score distributions -- how it scores
// positives, and how it scores negatives -- as counts over 10 score bins. Then
// change only how many negatives exist. The ranking quality is untouched.
// Deterministic bin counts, so this prints the same numbers every run.

const BINS = 10;                                    // score bin 0 (low) .. 9 (high)
const POS_SHAPE = [1, 2, 3, 5, 8, 12, 16, 20, 18, 15];   // a decent model:
const NEG_SHAPE = [30, 22, 16, 11, 8, 5, 4, 2, 1, 1];    // positives skew high

function scale(shape, total) {
  const sum = shape.reduce((a, b) => a + b, 0);
  return shape.map((c) => Math.round((c / sum) * total));
}

/** Sweep the threshold from "flag nothing" down to "flag everything". */
function curve(pos, neg) {
  const P = pos.reduce((a, b) => a + b, 0);
  const N = neg.reduce((a, b) => a + b, 0);
  const pts = [];
  let tp = 0, fp = 0;
  for (let b = BINS - 1; b >= 0; b--) {
    tp += pos[b];
    fp += neg[b];
    pts.push({
      bin: b,
      tp, fp,
      tpr: tp / P,                  // = recall
      fpr: fp / N,
      precision: tp + fp === 0 ? 1 : tp / (tp + fp),
      flagged: tp + fp,
    });
  }
  return { pts, P, N };
}

/** ROC-AUC by the trapezoid rule -- the standard, and correct here. */
function rocAuc(pts) {
  let auc = 0, prevFpr = 0, prevTpr = 0;
  for (const p of pts) {
    auc += (p.fpr - prevFpr) * (p.tpr + prevTpr) / 2;
    prevFpr = p.fpr; prevTpr = p.tpr;
  }
  return auc;
}

/** Average precision: sum over thresholds of (recall_i - recall_{i-1}) * precision_i.
 *  This is what scikit-learn's average_precision_score computes, and it is NOT
 *  the trapezoid -- interpolating between PR points is optimistic. */
function avgPrecision(pts) {
  let ap = 0, prevRecall = 0;
  for (const p of pts) {
    ap += (p.tpr - prevRecall) * p.precision;
    prevRecall = p.tpr;
  }
  return ap;
}

const SCENARIOS = [
  { label: "balanced   ", pos: 1000, neg: 1000 },
  { label: "1 in 100   ", pos: 1000, neg: 99000 },
  { label: "1 in 10,000", pos: 1000, neg: 9999000 },
];

console.log("same model, same ranking, only the negative count changes");
console.log("");
console.log("prevalence     positives    negatives   ROC-AUC   PR-AUC   PR baseline   lift");
for (const s of SCENARIOS) {
  const pos = scale(POS_SHAPE, s.pos);
  const neg = scale(NEG_SHAPE, s.neg);
  const { pts, P, N } = curve(pos, neg);
  const base = P / (P + N);
  const ap = avgPrecision(pts);
  console.log(
    "  " + s.label,
    String(P).padStart(10),
    String(N).padStart(12),
    rocAuc(pts).toFixed(4).padStart(9),
    ap.toFixed(4).padStart(8),
    base.toFixed(5).padStart(13),
    (ap / base).toFixed(1).padStart(6) + "x"
  );
}

console.log("");
console.log("ROC-AUC is identical to four decimals in all three rows. It has to be:");
console.log("TPR is normalised by all positives and FPR by all negatives, so both");
console.log("axes are within-class rates and neither knows the class ratio.");
console.log("Precision is normalised by PREDICTED positives, so it feels every");
console.log("extra false alarm the larger negative pool produces.");
console.log("");

// What the numbers mean to whoever has to act on the alerts.
console.log("the operating point, at the threshold that catches ~80% of positives");
for (const s of SCENARIOS) {
  const pos = scale(POS_SHAPE, s.pos);
  const neg = scale(NEG_SHAPE, s.neg);
  const { pts } = curve(pos, neg);
  const op = pts.find((p) => p.tpr >= 0.8) || pts[pts.length - 1];
  const ratio = op.fp / Math.max(op.tp, 1);
  console.log(
    "  " + s.label,
    " recall", (op.tpr * 100).toFixed(0) + "%",
    " precision", ((op.precision * 100).toFixed(1) + "%").padStart(6),
    " flags", String(op.flagged).padStart(7),
    " wrong", String(op.fp).padStart(7),
    " -> " + ratio.toFixed(ratio < 10 ? 2 : 0).padStart(4) + " false alarms per catch"
  );
}
console.log("");
const liftOf = (s) => {
  const { pts, P, N } = curve(scale(POS_SHAPE, s.pos), scale(NEG_SHAPE, s.neg));
  return avgPrecision(pts) / (P / (P + N));
};
console.log("Watch the lift column too. It goes UP as the problem gets harder:");
console.log(
  liftOf(SCENARIOS[0]).toFixed(1) + "x when balanced,",
  liftOf(SCENARIOS[2]).toFixed(1) + "x at 1 in 10,000. Lift over a random"
);
console.log("baseline is not a measure of usefulness -- the baseline is collapsing");
console.log("faster than the model is. \"10x better than random\" can be unusable.");
console.log("");
console.log("Report ROC-AUC = 0.92 in all three rows and you are telling the truth");
console.log("about ranking and nothing about the job. At 1 in 10,000 the same model");
console.log("buries every real case under a pile of false ones, and the only");
console.log("number that showed you was precision.");

Two other places the tabs diverge, both worth carrying into an interview. JavaScript's pts.find((p) => p.tpr >= 0.8) becomes next((p for p in pts if p["tpr"] >= 0.8), pts[-1]) — next on a generator with a default is Python's find, and unlike find it makes the fallback explicit rather than hiding it behind || pts[pts.length - 1]. And the dynamic precision in ratio.toFixed(ratio < 10 ? 2 : 0) becomes a nested f-string replacement field, f"{ratio:.{2 if ratio < 10 else 0}f}", which is the one piece of f-string syntax people are routinely surprised to learn exists.

The middle table is the one to remember. Same model, same recall, and precision goes 86% → 5.9% → 0.1%. At one-in-ten-thousand prevalence, catching 81% of the positives means shipping 1,605 false alarms for every real one — a queue no team will work and no user will tolerate. ROC-AUC reported 0.9092 for all three.

The 1-in-100 row, as the people downstream experience it
stagevolumestep
Cases scored 100,000 —
Flagged by the model 13,680 14%
Actually positive 810 5.9%
volume reaching this stage

This is the same model whose ROC-AUC is 0.9092. It flags 13,680 of 100,000 cases and 810 of those are real — precision 5.9%, or sixteen false alarms for every catch. Recall is still 81%: the model's ranking is genuinely good and the queue is still mostly noise. The right-hand column is the number a reviewer feels, and no AUC on either curve would have told you it.

So which one do you report

SituationReport
Roughly balanced classes, both errors matter similarlyROC-AUC
Rare positives (under ~10%), the positive class is the pointAverage precision / PR-AUC, plus precision and recall at your actual threshold
You need to choose an operating pointPrecision at the recall you need — not an AUC at all
Comparing two models' ranking quality on the same dataEither, consistently; ROC-AUC is the more common convention

The honest answer to "which is better" is report both, and report the operating point, because they answer different questions. ROC-AUC summarises ranking across every threshold. Average precision summarises the minority-class trade-off. Neither tells anyone what happens when you ship, and the number that does is precision and recall at the threshold you actually picked.

The threshold is a cost decision, not a modelling one

0.5 is a default, not an answer. The threshold is where you convert a ranking into behaviour, and the right value follows from what each mistake costs.

  • A false positive is expensive — a blocked legitimate payment, a wrongly flagged account, an analyst hour. Raise the threshold; take less recall.
  • A false negative is expensive — a missed fraud, a missed diagnosis, a churned customer you could have saved. Lower it; accept the false alarms.
  • You have a fixed capacity — the review team can handle 500 cases a day. Then the threshold is determined by capacity, not by either cost, and the metric that matters is precision in the top
  1. Say so.

What's next

Two lessons in and the pattern is the same both times: a number was correct and the conclusion drawn from it was not, because of something the number structurally could not see. The last lesson in this module does it to an analysis rather than a model — a retention query that reports a collapse where nothing happened at all.