Cost and Latency in Production AI Systems

A demo serves one user, slowly, and nobody sees the bill. Production serves thousands, on a latency budget, and finance sees the bill every month. The gap between those two worlds is where a lot of AI deployments quietly die, and closing it is squarely the FDE's job. This page is about the two numbers that decide whether a working prototype becomes a viable product: dollars per request and time to a response.

Why this is the job

The pattern: the pilot delights fifty users, the customer says "roll it out to fifty thousand," and now the same design costs a hundred times as much and the shared model is slow under load. The FDE is the one who saw it coming and designed the levers in. An engineer who can't discuss cost per request and p99 latency in production terms hasn't operated a real deployment, and interviewers know it.

Before reading on: your prototype makes one large-model call per user question and it's wonderful. You're asked to support 100× the traffic on the same budget. Name two levers before you reach for "a bigger cluster."

The two that move the most: don't call the big model when you don't have to (cache repeats, route easy questions to a small model), and don't send more tokens than you need (trim the prompt and the retrieved context). Neither is infrastructure; both are design.

The cost levers

• Cache the repeats. Many production workloads have a heavy head of near-identical requests. A response cache on exact or semantically-close repeats removes the model call entirely for the cheapest possible request — the one you don't make.
• Right-size the model. Most questions are easy; a few are hard. Route the easy majority to a small, cheap model and reserve the large model for the hard tail. A fast classifier in front pays for itself many times over.
• Trim the tokens. Cost is linear in tokens, so over-fetched retrieval context is a tax on every request. Retrieve fewer, more relevant passages (this also cuts latency and improves groundedness — the same move helps three problems).
• Cache the shared prefix. When every request shares a long system prompt or conversation prefix, caching its computed state avoids re-processing the identical part on each call.

Real anchors, dated

The method needs real numbers to bite on, so here are the list prices for two vendors' flagship and small models. Prices are volatile — re-check the linked pages before quoting them to an interviewer or a customer.

Two things to read off the table beyond the absolute figures. Output tokens cost a multiple of input tokens on every row, which makes verbose answers a cost lever in their own right. And each vendor's small model runs at a fraction of its flagship's price, which is what turns the routing lever from tidying into real money: every easy question routed down removes most of that request's cost.

Prompt caching and batch, priced

The "cache the shared prefix" lever has vendor-published economics. On Anthropic's pricing, writing a prefix into the five-minute cache costs 1.25 times the base input price and a later cache hit costs 0.1 times — the same page states that the five-minute tier "pays off after just one cache read". OpenAI prices cached input at one-tenth of standard input. The payback logic:

Anthropic five-minute cache; prices relative to base input = 1.0
  cache write = 1.25        cache hit = 0.10
  2 requests, uncached:  1.0  + 1.0  = 2.0
  2 requests, cached:    1.25 + 0.10 = 1.35   → ahead after a single hit
  10 requests, cached:   1.25 + nine hits at 0.10 each = 2.15
                         vs 10.0 uncached — most of the prefix cost gone

The caveat that keeps this honest: the saving applies to the shared prefix only, the prefix must be byte-identical between requests, and the entry expires, so the economics depend on how often the same prefix recurs within the cache lifetime. One more published lever: both vendors sell batch processing — asynchronous, non-interactive — at a 50% discount from Anthropic and 50% from OpenAI, the cheapest option for anything that can wait, evals included.

The latency levers

• Stream the tokens. The single biggest perceived-latency win: emit tokens as they're generated so the user sees a response start immediately. Time-to-first-token becomes the number that matters, and it's far smaller than time-to-complete.
• Cache the prefix. The same prefix cache that saves cost also cuts the time spent re-processing a long shared prompt, which shrinks time-to-first-token directly.
• Use a smaller or faster model where you can. The routing move that saves cost usually saves latency too — small models are quicker.
• Parallelize the pipeline. Retrieval, guardrail checks, and any tool calls that don't depend on each other should run concurrently, not in a chain.

The tension to name out loud: cost, latency, and quality pull against each other. A bigger model is more accurate and slower and dearer; aggressive caching is cheap and fast and risks staleness. What earns credit is saying which you're optimizing and why — tied to what would actually break the deployment.

Latency anchors, dated

The cost half of this page anchored its method to real list prices; the latency half needs anchors too. Two numbers describe how a model feels to use: time to first token (TTFT) and output speed. Artificial Analysis, the benchmark most people quote, defines them precisely: TTFT is "The time in seconds between sending a request to the service or system and receiving the first token of the response", and output speed is "The average number of tokens received per second, after the first token is received." TTFT is the silence a user stares at; output speed decides how long the answer takes to finish once it has started.

Two measured anchors, one per vendor's small model:

Read the rows against each other: gpt-5.4-mini streams faster once it starts, yet a user waits over six seconds before anything appears — that first-token figure reflects a reasoning-mode variant spending thinking time before it answers, the same effect the frontier caution below describes. Which model "feels faster" depends on which number your flow exposes. These benchmark figures move continuously — new model revisions, new serving hardware — so quote them with an access date, as this table does.

Frontier reasoning models need a separate caution, and no single TTFT figure does them justice: a reasoning mode spends thinking time before the first answer token, which can push the user-visible wait far beyond small-model TTFT. Artificial Analysis tracks this separately as "Time to First Answer Token". Design user-facing flows around that difference — a flow tuned for a second of silence needs a progress indicator, an interim status message, or a smaller model when the thinking stretches the wait to something much longer.

Both vendors publish which lever to pull. OpenAI's latency-optimization guide quantifies an asymmetry worth memorizing: cutting output tokens by half reduces latency by roughly half, while cutting input tokens by half improves latency by only roughly one to five percent. Output length is the lever, which makes "answer concisely", a maximum-output cap, and terse structured formats latency work as much as cost work. Anthropic's guidance is qualitative and points the same way: pick the fastest model that is adequate ("For speed-critical applications, Claude Haiku 4.5 offers the fastest response times while maintaining high intelligence"), constrain output length, and stream.

A p99 budget, worked

A latency target becomes engineering only when it is decomposed per stage. The guarded pipeline from the observability trace — retrieval, guardrail pre-check, model call, groundedness check — is a chain, and the user's wait is the sum of whatever sits on the blocking path. Budget it at two points of the distribution: p50 for the typical request, p99 for the tail the support queue hears about.

Illustrative end-to-end budget — every figure in this box is invented;
the method is what to carry. Answer length 300 tokens.

stage                          p50          p99
retrieval                       90 ms       450 ms
guardrail pre-check             15 ms        40 ms
model: time to first token     800 ms     2,500 ms
model: generation              300 tokens ÷ 60 tokens/second = 5,000 ms
groundedness check             400 ms     1,800 ms

first visible token, streaming (retrieval + pre-check + TTFT):
  p50:   90 + 15 + 800                    =   905 ms
  p99:  450 + 40 + 2,500                  = 2,990 ms

full response, groundedness check blocking the stream:
  p50:   90 + 15 + 800 + 5,000 + 400      = 6,305 ms
  p99:  450 + 40 + 2,500 + 5,000 + 1,800  = 9,790 ms

Two lessons are folded into the two totals. First, users feel time to first visible token, so streaming hides nearly all of the generation time: the same pipeline feels like under a second or like more than six, depending on whether tokens reach the screen as they are produced. Second, the groundedness check breaks that trick. As the guardrails page notes, the judge runs in sequence, after the full draft and before the user sees anything, so a blocking check turns the streamed experience back into the full-response wait. That leaves three designs: make the check fast enough to fit the budget (a small judge model over a short answer), run it on a sampled or risk-routed basis instead of on every request, or stream the draft and retract it if the check fails — tolerable only where a briefly visible wrong answer is survivable, which in a regulated deployment it often is not.

One caveat keeps the arithmetic honest: summing each stage's p99, as the box does, gives a conservative planning bound rather than the true p99, because the slowest one percent of retrievals and the slowest one percent of model calls mostly happen on different requests. In practice the end-to-end tail is dominated by whichever stage has the fattest tail of its own. So budget per stage, measure per stage — the same per-stage timings the observability trace records — and read the real end-to-end distribution from production, where the spreadsheet's estimate gets corrected.

The ceilings: rate limits

Capacity runs out before price does: every provider caps throughput per account, so a rollout can hit a hard ceiling while the bill is still comfortable. Anthropic's limits are set per usage tier and measured three ways per model class: requests per minute, input tokens per minute (ITPM), and output tokens per minute (OTPM). Exceeding any of them returns "a 429 error" with a retry-after header, and enforcement uses a token bucket algorithm, so capacity replenishes continuously rather than resetting on a clock boundary. The same page ties this chapter's caching lever to capacity as well as cost, with its own worked example: "With a 2,000,000 ITPM limit and an 80% cache hit rate, you could effectively process 10,000,000 total input tokens per minute" (Anthropic rate limits). The prefix cache you priced earlier also buys headroom under the ceiling.

OpenAI's limits have the same shape: requests and tokens per minute or per day, applied at the organization and project level, with usage tiers that graduate automatically as spend accumulates.

The FDE lesson: launch-day arithmetic includes the ceiling. Peak concurrent users times tokens per request, held against the tokens-per-minute limit, tells you whether the rollout fits the current tier before the first user signs in — and the retry-with-backoff path for a 429 belongs in the design from the start.

Self-hosting: the crossover question

An interviewer who follows the cost analysis will push one step further: when does self-hosting an open-weights model beat paying the API? The reasoning matters more than any figure. An API provider amortizes its accelerators across many tenants, so it buys a level of utilization a single customer rarely has; a self-hoster pays for idle capacity around their own traffic peaks, and pays again in operations labor to keep inference serving healthy. Self-hosting therefore tends to win in three situations: sustained volume high enough to keep the hardware busy, a custom fine-tuned model the APIs cannot serve, or data-residency and compliance constraints that rule the API out entirely (Security, Compliance & Data covers the third). And say plainly what the numbers won't say for you: credible public crossover figures do not exist — survey claims range over two orders of magnitude — so anyone quoting a break-even owes you their assumptions: volume, utilization, hardware, and labor. Naming the drivers and then demanding those assumptions is the strong interview answer.

Bad / Good / Great — "make it work at scale"

Bad — "we'd scale horizontally, add more instances." You treated an AI system like a stateless web service. The cost is per token and the latency lives in the model call, so extra instances leave both exactly where they were — you've added servers around a bottleneck they can't touch.

Good — "cost is tokens times volume, latency is time-to-first-token; I'd cache and use a smaller model where I can." Correct frame and real levers — solid mid-level. The gap: no sense of which term dominates or which constraint is closest to breaking.

Great — "I'd measure cost per request and p99 first, find the dominant term, then attack it: cache the head, route the easy majority to a small model, trim over-fetched context, and stream for perceived latency. I'd instrument both per request from day one, because at scale a small regression is a large bill or a support queue. And I'd optimize whichever is closest to breaking the deployment first, ahead of whichever is easiest to reach for." You turned "it scales" into named levers on measured numbers, with a priority rule.

What to carry into the interview

In the system design round, "it scales" is a non-answer. Name the two numbers — dollars per request, p99 latency — say which one is closer to breaking the deployment, and give concrete levers on the dominant term. Lead with the measurement and the priority it sets.

Related: Observability & Debugging — you can only optimize the number you're watching.

Next: Security, Compliance & Data — the constraints that decide whether the system is allowed to exist at all.