Inference and serving, from tokens per second to cost per request
Why prefill and decode hit different limits, how the KV cache caps concurrency, which serving techniques move which number, and the latency, memory and cost arithmetic interviewers expect you to do out loud.
What you'll be able to do
- Explain why prefill is compute-bound and decode is memory-bandwidth-bound, and tie each to time to first token and time per output token
- Calculate the KV cache for a concrete model shape and turn a GPU memory budget into a concurrency limit
- Match continuous batching, paged attention, quantisation, speculative decoding and prefix caching to the bottleneck each one relieves
- Work out cost per request and the self-host versus API break-even from stated assumptions
- Design the client side of an LLM call with retries, layered timeouts, fallbacks and a degradation plan
Before this: how-llms-work
Log in to read the rest of this lesson
Why prefill and decode hit different limits, how the KV cache caps concurrency, which serving techniques move which number, and the latency, memory and cost arithmetic interviewers expect you to do out loud. This walkthrough runs about 18 minutes, with runnable code you can edit and re-run as you read. A new account gets 7 days of Pro free, no card required.
Log in to continueThe first module of every track is free to read on the web: see what's open in AI interviews.