1. Live coding: build the prompt for the next turn under a fixed token budget.
The full question
Live coding: build the prompt for the next turn under a fixed token budget. The catch is our prefix cache.
Model answer
The trap is the obvious implementation: keep the last N tokens and drop the oldest. That truncates from the front, which changes the prefix on every turn and destroys the inter-turn cache from Q2 - you convert a 95% hit rate into near zero and multiply serving cost.
The correct shape orders the prompt by stability, so everything that changes lives at the end:
def build_context(char, memory, turns, user_msg, budget, tokenize):
# Stable prefix: identical across turns and often across users.
prefix = char.system_prompt + char.definition + char.example_dialogue
head = tokenize(prefix)
# Semi-stable: rolling summary plus durable facts. Changes only at
# a compaction event, never on every turn.
mem = tokenize(memory.summary + memory.facts)
tail = tokenize(user_msg)
remaining = budget - len(head) - len(mem) - len(tail) - RESERVED_FOR_REPLY
if remaining < 0:
raise ContextTooSmall
# Fill backwards from the most recent turn, but emit in order.
kept, used = [], 0
for t in reversed(turns):
tt = tokenize(t)
if used + len(tt) > remaining:
break
kept.append(tt)
used += len(tt)
kept.reverse()
return head + mem + flatten(kept) + tail, len(head) + len(mem)
Two things to say out loud. First, return the stable prefix length so the caller knows where the cache boundary sits. Second, compaction must be event-driven, not per-turn: when the recent-turn window overflows, summarise a whole block at once and accept one cache invalidation, rather than shifting the boundary every turn. Amortised, that is one expensive prefill every dozens of turns instead of one every turn.
Worth sketching. Ordering by stability is the entire design.
flowchart LR
A["Pinned prefix<br/>persona and definition"] --> B["Rolling summary<br/>and durable facts"]
B --> C["Recent turns verbatim"]
C --> D["Current user message"]
D --> E["Reply budget reserved"]Follow-ups: Where do you insert retrieved older turns without breaking the cache boundary? How would you unit-test that a refactor did not silently drop the hit rate?