How to Set a Latency Budget for AI Features

Most teams pour weeks into model quality and never write down a single number for speed. They tune the prompt, swap to a smarter model, argue about evals — and ship a feature that technically works but feels slow enough that people stop using it. Quality you can see in a demo. Latency you only feel in production.

It's not a niche worry either. In LangChain's survey of 1,340 practitioners, latency was the second-biggest barrier to putting AI agents into production, named by 20% of respondents (LangChain, 2025). The fix isn't a faster model. It's deciding, up front, how slow your feature is allowed to be — and engineering against that ceiling. This guide covers how to set a latency budget and actually hit it.

Key Takeaways

• Latency is the #2 barrier to shipping AI agents to production, cited by 20% of 1,340 practitioners (LangChain, 2025) — yet most teams never set a target.
• A latency budget is a hard p95 ceiling per interaction, derived from human perception limits (instant ≈100ms, flow ≈1s).
• Time-to-first-token is the number users actually feel — stream early, cache off the hot path, and gate releases on p95 so the budget never quietly regresses.

What is a latency budget for an AI feature?

A latency budget is a hard ceiling on how long an interaction may take — measured at p95, not on average — and every part of the request path has to fit inside it. Think of it like a performance budget for page weight, but for time. You set the number first, then spend it.

Where does the number come from? Human perception, not vibes. Jakob Nielsen's response-time limits still hold: about 0.1 second feels instantaneous, 1 second keeps a user's flow of thought unbroken, and past 10 seconds you've lost their attention entirely (Nielsen Norman Group, 1993). Google's RAIL model sharpens the top end — respond to a user input within 100ms and the interface feels immediate (web.dev, 2020).

So a budget isn't one global number. A chat reply that streams can run longer than an autocomplete that must feel instant. You assign each interaction a tier, pick the p95 ceiling for that tier, and treat it as a constraint the design has to satisfy — the same way you'd treat a memory limit.

Why does latency decide whether AI features get used?

Because speed and usage are tied together tightly, and the data on it is brutal. Every fraction of a second you add costs you engagement, and every one you cut buys some back. This is the oldest result in web performance, and it didn't stop applying when the backend became an LLM.

Deloitte's "Milliseconds Make Millions" study found that a 0.1-second improvement in mobile load time lifted retail conversions 8.4% and average order value 9.2% (Deloitte, 2020). The downside is just as steep: 53% of mobile visitors abandon a page that takes longer than three seconds (Akamai/Google, 2020). An AI feature buried behind a three-second spinner inherits that abandonment, no matter how good its output.

Our take: The tier that breaks teams is "Responsive." They assume a chat feature has seconds to spare, so they skip streaming — and a 4-second silent wait lands in the "Lost" column even though the total generation was reasonable.

Where do the milliseconds actually go?

They scatter across a path most teams never decompose. A single AI request is auth, then a prompt/config read, then time-to-first-token from the model, then token streaming, then any post-processing. Each is a line item in your budget, and the model is rarely the only one that matters.

Time-to-first-token (TTFT) is the gap between sending the request and the first token coming back — and for fast, non-reasoning models it lands between roughly 0.3 and 1.2 seconds, while throughput swings about tenfold across models (Artificial Analysis, 2025). Reasoning models add seconds of think-time before a single visible token. That choice alone can blow a "Responsive" budget on its own.

The number to govern is p95, or even p99 — not the mean. Averages hide the tail, and the tail is what users complain about. A feature with a 600ms average and a 6-second p99 feels broken to one user in a hundred, every hundred requests. Budget the tail.

How do you design an AI feature to hit its budget?

You attack the segments you own and disguise the ones you don't. Most of the perceived win comes from a handful of moves, and none of them require a smarter model.

Stream tokens. TTFT is the number users feel, not total completion time — first token on screen in under a second reads as fast even when the full answer takes five. Then get your own segments off the hot path. Validating an API key against the database on every request, or fetching a prompt from a slow store, adds latency you fully control. Cache both: a validated key in memory and a prompt at the edge turn a round-trip into a near-zero read. We go deeper on the read side in our guide to prompt caching for LLM cost.

Where the read happens matters too. Cloudflare Workers effectively eliminate cold starts by spinning up a V8 isolate during the TLS handshake — versus serverless cold starts that range from ~100ms to multiple seconds (Cloudflare, 2020). Serving prompts and config from a global edge cache, instead of a single origin region, shaves the network segment for users far from your database. Parallelize independent work — retrieval, tool calls, and auth shouldn't run in series if they don't depend on each other.

Our finding: When we moved prompt reads off the origin database and served them from an edge cache, the read segment dropped from a cross-region round-trip to single-digit milliseconds — the origin became a fallback, not the hot path. The model didn't change; the budget did.

Cutting latency and cutting cost usually pull in the same direction here — the same caching that speeds reads also trims spend, which we cover in how to reduce LLM costs.

How do you keep the budget from regressing?

You measure it in production and make it a release gate. A latency budget that lives in a design doc is decoration. The one that lives in your alerting is a constraint, and only the second kind survives contact with shipping.

Instrument p95 and p99 TTFT for each AI interaction and alert when either crosses its tier ceiling — not the average, which will lie to you. Then tie latency to the thing most likely to move it: the prompt. A one-line prompt edit can quietly double output length, and therefore time, with no error to warn you. Treating each prompt change as a tracked, measured release is the same discipline we describe in prompt versioning best practices — and latency is just another metric you gate on before widening a rollout.

Frequently Asked Questions

Isn't a faster model the real fix for latency?

Rarely on its own. TTFT for fast non-reasoning models already sits near 0.3–1.2s (Artificial Analysis, 2025), so the segments you control — auth, prompt reads, cold starts, serial work — are often where the budget actually leaks. Streaming and caching usually beat a model swap.

Should I budget against average latency or p95?

p95, and watch p99. Averages hide the tail, and the tail is what users feel. A feature averaging 600ms with a 6-second p99 reads as broken to roughly one request in a hundred. Since 53% of visitors abandon after three seconds (Akamai/Google, 2020), the slow tail is the part that costs you.

What's a good latency target for a chat feature?

Aim for first token under one second. Nielsen's limits show that one second keeps a user's flow of thought unbroken (Nielsen Norman Group, 1993). Total completion can run longer if you stream, because perceived speed tracks TTFT, not the final token — so stream early and show progress.

Does serving prompts from the edge really matter for latency?

For users far from your origin, yes. Cloudflare Workers avoid cold starts entirely, while serverless cold starts can run from ~100ms to seconds (Cloudflare, 2020). Reading a prompt from a nearby edge cache instead of a single-region database removes a network round-trip from every request.

The takeaway

The teams whose AI features feel fast aren't running secret models — they're spending a number on purpose. They set a p95 ceiling per interaction, decompose the request path, and defend the segments they own: auth, prompt reads, cold starts, and the moment the first token hits the screen.

Start with one feature. Pick its perception tier, write down the p95 budget, and instrument TTFT in production. Stream tokens so the wait disappears, move prompt and key reads off the hot path, and gate every prompt change on latency the way you'd gate on quality. Do that and speed stops being something you hope for and becomes something you've already paid for.

PromptVault serves prompt reads from a global edge cache and keeps the origin as a fallback — so fetching the prompt never eats your latency budget. See how it works.