AI interviews · Running LLM systems, evals, serving and safety

Inference and serving, from tokens per second to cost per request

Why prefill and decode hit different limits, how the KV cache caps concurrency, which serving techniques move which number, and the latency, memory and cost arithmetic interviewers expect you to do out loud.

18 min read Full lesson in the app Patterns: prefill-vs-decode, kv-memory-caps-batch, stable-prefix-first, goodput-not-throughput

What you'll be able to do

Before this: how-llms-work

Log in to read the rest of this lesson

Why prefill and decode hit different limits, how the KV cache caps concurrency, which serving techniques move which number, and the latency, memory and cost arithmetic interviewers expect you to do out loud. This walkthrough runs about 18 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.

Log in to continue

The first module of every track is free to read on the web: see what's open in AI interviews.