1. Practical coding: write a service that proxies streaming TTS to a browser and cancels cleanly when the user navigates away.
Model answer
The requirement worth surfacing before writing code: a disconnected client must cancel the upstream generation. Otherwise you keep paying for characters nobody hears, and at agent scale abandoned streams become a real cost line.
import asyncio, contextlib
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
app = FastAPI()
@app.post("/speak")
async def speak(request: Request):
body = await request.json()
upstream = await tts.open_stream(body["text"], body["voice_id"])
async def audio():
played = 0
try:
async for chunk in upstream:
if await request.is_disconnected():
break
played += len(chunk)
yield chunk
except asyncio.CancelledError:
raise
finally:
with contextlib.suppress(Exception):
await upstream.close() # idempotent, stops billing
await meter.record(body["request_id"], played)
return StreamingResponse(audio(), media_type="audio/mpeg")
Points to say out loud. The finally block is the contract: close upstream on disconnect, on exception and on normal completion, and make close idempotent. Metering on bytes actually delivered, not characters submitted, is the honest choice and it is also what you need for the transcript truncation problem in a live agent. Do not buffer the whole response to set Content-Length; chunked transfer with flush per chunk is the point. Disable proxy buffering explicitly or your carefully optimised time-to-first-audio disappears into nginx.
For an agent rather than a one-shot request, upgrade this to a WebSocket so successive text chunks reuse the same conditioned stream, and add a client-to-server cancel message so barge-in does not have to rely on TCP teardown, which can take seconds to notice.
Follow-ups: How do you propagate a mid-stream upstream failure to a client that already received a 200 and some audio? Where would you add a cache, and what is the cache key for a voice plus text pair?