An AI feature can have a two-second average and still feel slow, because the requests users remember are the ones that take eight. Latency work starts with a budget and a trace, not with a smaller model. If you do not know which span is slow, you are guessing.
Write the budget before you optimise
Pick a target for the interaction, not the model call. A search answer might need to feel instant; a long-form draft can take longer if it streams. Then allocate the budget across the request path. A rough example for a 3-second p95 target:
| Span | Budget | Typical fix |
|---|---|---|
| Queue / admission | 100 ms | Concurrency limit, priority queue |
| Auth + tenant lookup | 50 ms | Cache, parallelise with retrieval |
| Retrieval | 400 ms | Hybrid index, smaller top-k, rerank fewer |
| Time to first token | 800 ms | Prompt cache, smaller model, shorter input |
| Generation | 1,200 ms | Stream, cap output tokens |
| Tools (if any) | 300 ms | Parallel calls, timeouts, cached reads |
| Network + render | 150 ms | Compress, avoid layout shift |
The numbers are illustrative. The important part is that they add up and every span has an owner.
Measure time to first token separately
Streaming changes the perceived latency curve. The user sees something at the first token, even if the full answer takes six seconds. Track time to first token and total time separately. A slow first token usually means a large prompt, a cold cache or a queue. Slow completion usually means too many output tokens or a slow tool.
Parallelise what does not depend on the model
Authentication, tenant lookup, retrieval and prompt assembly often run in sequence because the code grew that way. Run independent work concurrently. Fetch the user's permissions and the candidate documents at the same time, then assemble the prompt once both are ready.
permissions, candidates = await asyncio.gather(
load_permissions(user_id),
retrieve(query, tenant_id),
)
prompt = assemble(candidates, permissions)Cut the input before you cut the model
A smaller model is not automatically faster if the prompt is enormous. Trim retrieved context, remove duplicated instructions, and cache the static prefix. Many teams find that a 40% smaller prompt improves both latency and answer quality because the model has less to ignore.
- Cache the system prompt and tool schemas where the provider supports it.
- Send only the top few reranked passages, not twenty.
- Summarise conversation history instead of replaying all of it.
- Cap output tokens; long answers are often padding.
Fail fast and degrade gracefully
A slow dependency should not hold the whole request until the client gives up. Set explicit timeouts per span. If retrieval times out, answer from the model's general knowledge with a visible caveat. If a tool times out, return a partial result and a retry action. A fast, honest partial answer beats a spinner.
A latency budget is a product decision written down. Without it, every optimisation is a guess.
What to remember
- Set an end-to-end p95 target and allocate it across queue, auth, retrieval, model, tools and render.
- Measure time to first token separately from total generation time.
- Run independent work concurrently; do not serialise auth, retrieval and prompt assembly.
- Reduce input and cap output before switching to a weaker model.
- Add per-span timeouts and a deliberate degraded response.
