Anthropic Interview: Design an LLM Inference API with GPU Batching

Part of the Anthropic system design question bank. The question is representative of the round. The analysis and solution are this site's own.

Problem statement

The service sits between API clients and a fleet of GPU machines. It accepts generation requests, decides which requests run together on which GPUs, streams tokens back, and protects itself when demand exceeds capacity. Training, fine-tuning, and billing systems are out of scope. Safety checks are in scope only as a stage in the request path.

A web service with a thread per request is the naive answer, and it wastes almost all of the GPU. The rest of the design is about why, and what replaces it.

Clarifying questions

Several parameters shape the design. Each question below settles one, with the assumption this solution uses.

• Streaming or not? Both. Chat clients stream tokens as they arrive. Batch jobs want one response.
• One model or several? A few models, each with more than one version live during rollouts.
• What are the latency targets? Two numbers, because they have different causes. Time to first token (TTFT) covers queueing and prompt processing. Time per output token (TPOT) covers each generation step.
• What traffic tiers exist? Paid and free. Paid traffic is protected first under overload.
• What does traffic look like? Bursty. A launch or a viral product can double demand in minutes.
• How long are prompts and answers? For practice: prompts average 1,500 tokens, answers 300 tokens, and some prompts reach 100,000 tokens.

What makes LLM serving hard

An inference request is not like a web request. Three facts make it different.

First, the work per request is unknown at admission. The server knows the prompt length, but nobody knows how many tokens the answer will take until the model stops.

Second, generation runs one token at a time. Each new token needs a full pass through the model. A 300-token answer is 300 sequential passes.

Third, each of those passes is limited by memory bandwidth. The GPU must read every model weight from memory to produce one token for one sequence, and then it does very little arithmetic with them. Alone, one sequence leaves most of the GPU's compute idle.

So the driving tension is throughput versus latency. Put many sequences in each pass and the GPU does far more useful work per weight read, which makes each token cheap. But bigger batches make every step slower, and requests wait longer to join. Every design choice below moves along that line.

flowchart LR
  C([Clients]):::user --> G[Gateway]:::svc
  G --> S[Scheduler<br/>forms batches]:::svc
  S --> R[GPU replica<br/>model + KV cache]:::store
  R -->|tokens| G
  G -->|stream| C
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

Key idea. LLM serving is a scheduling problem. The GPU is fast only when many sequences share each pass, so the scheduler decides both the cost and the latency.

Key concepts

This section covers the ideas the design depends on. Read it before the requirements if any term is new.

Prefill and decode

Generation has two phases. Prefill processes the whole prompt in one pass. All prompt tokens go through the model in parallel, so prefill does a lot of arithmetic per weight read. It is compute-bound, and it produces the first output token.

Decode then produces the remaining tokens one pass at a time. Each pass handles one new token per sequence. It reads all the weights to do a small amount of math, so decode is memory-bandwidth-bound. Most of a request's wall-clock time is spent in decode.

flowchart LR
  P["Prefill: 1,500 prompt tokens<br/>one pass, compute-bound"]:::hot --> T1["token 1"]:::svc
  T1 --> D["Decode: one pass per token<br/>memory-bound, repeated 299 times"]:::svc --> TN["token 300"]:::svc
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef hot fill:#fbe9e4,stroke:#c4492d,color:#171717;

The KV cache

Attention needs, for every earlier token, a key vector and a value vector from every layer. Recomputing them at each step would repeat all prior work. So the server keeps them in GPU memory for the life of the request. This is the KV cache.

The KV cache grows by one entry per layer for every token, and it lives until the request finishes. After the weights, it is the largest user of GPU memory. In practice it decides how many sequences fit in a batch, which makes it the real limit on throughput.

Batching strategies

Continuous batching, also called iteration-level scheduling, is the baseline in modern serving engines. It was introduced in the Orca paper and is standard in open-source engines today.

Paged KV memory

If the server reserves KV memory for the maximum possible length of every request, most of that memory stays empty and few sequences fit. Paged allocation splits KV memory into small fixed blocks, such as 16 tokens each. Each sequence has a block table, like a page table in an operating system. Memory grows only as the sequence grows. Requests that share a prefix, such as the same system prompt, can point to the same blocks.

Key idea. Decode is memory-bound, so batching is the lever; the KV cache is memory, so it caps the batch. Continuous batching keeps the batch full, and paged memory makes the batch as large as memory allows.

1. Requirements

Before reading on. List the functional and non-functional requirements. Then name the one property you would protect at all costs and the one constraint that shapes the design.

1.1 Functional requirements

• Accept generation requests with a model, messages, sampling settings, and a maximum output length.
• Stream tokens to the client as they are produced, or return one response.
• Serve several models and versions at the same time.
• Enforce rate limits per API key and per tier.
• Report token usage for every request.
• Run safety checks on input and output.

1.2 Non-functional requirements

• Latency. For practice: TTFT under one second at p95 and TPOT under 50 ms at p95 for paid traffic.
• Throughput per GPU. As high as the latency targets allow, because GPUs are the dominant cost.
• Availability. The API keeps answering during GPU failures and bursts, possibly by rejecting some traffic quickly.
• Isolation. One customer's burst must not ruin latency for everyone else.

1.3 The constraint versus the property

Availability of the API is the property to protect. Under overload the system may reject requests, but it must reject them quickly and clearly, and paid traffic must keep flowing. GPU cost is the constraint that shapes the design. GPUs are scarce, slow to add, and expensive, so the design exists to extract maximum tokens per GPU-second inside the latency targets. Latency is the budget that cost is traded against.

Key idea. Protect availability with fast, tiered rejection. Design around GPU cost. Spend latency, within its targets, to buy throughput.

1. Back-of-the-envelope estimation

The numbers size three things: how many sequences fit on a replica, how fast a replica can generate, and how many GPUs the traffic needs. All inputs are for practice. Hardware figures come from the vendor's published specifications.

2.1 KV cache per token

KV bytes per token = 2 (key and value) × layers × KV heads × head size × bytes per number. For practice, take a 70-billion-parameter model with 80 layers, 8 KV heads, a head size of 128, and 16-bit numbers:

2 × 80 × 8 × 128 × 2 = 327,680 bytes, about 0.33 MB per token.

A conversation of 2,000 tokens needs about 650 MB of KV cache.

2.2 How many sequences fit

The weights take 70 billion × 2 bytes = 140 GB. A replica of 8 GPUs with 80 GB each has 640 GB. After weights and working memory, assume 400 GB is left for the KV cache.

400 GB / 327,680 bytes ≈ 1.22 million tokens of cache. At 2,000 tokens per sequence, about 610 sequences fit at once. That is the memory ceiling on batch size.

2.3 How fast one replica decodes

Each decode step must read all 140 GB of weights plus the KV cache of every sequence in the batch. An H100 SXM GPU reads memory at about 3.35 TB/s, so 8 of them read about 26.8 TB/s together.

With a batch of one, a step reads 140 GB: 140 / 26,800 ≈ 5.2 ms. That is about 190 tokens per second for one user, and 190 tokens per second for the whole replica.

With a batch of 256 sequences at 2,000 tokens each, the step also reads 256 × 2,000 × 327,680 bytes ≈ 168 GB of KV cache, about 6.3 ms more. The step takes about 11.5 ms. Each user still sees about 87 tokens per second, and the replica produces 256 × 87 ≈ 22,000 tokens per second.

That comparison is the whole argument for batching: 117 times the throughput for about twice the per-token latency. These are ceilings. Real engines lose some of this to scheduling, sampling, and communication between GPUs, so plan with a measured number.

2.4 GPU count for the traffic

For practice: a peak of 500 requests per second, with 1,500 prompt tokens and 300 output tokens each.

Decode needs 500 × 300 = 150,000 output tokens per second. Suppose a benchmark shows one replica sustains 9,000 tokens per second while it meets the TPOT target. That needs about 17 replicas, or 136 GPUs.

Prefill needs 500 × 1,500 = 750,000 prompt tokens per second. A forward pass costs about 2 × parameters floating-point operations per token, so 2 × 70 billion × 750,000 ≈ 1.05 × 10^17 operations per second. An H100 delivers about 989 teraflops of dense 16-bit compute. At 50% utilization, prefill needs about 212 GPUs.

Together that is about 350 GPUs. Add 30% for bursts and lost machines, and plan for about 57 replicas, or 456 GPUs.

Notice what the numbers say: with long prompts and short answers, prefill takes more GPUs than decode. That fact drives a deep dive in section 6.

2.5 Cost per token

Cost per million output tokens = (price per GPU-hour × GPUs per replica) / (tokens per second per replica × 3,600) × 1,000,000. Every factor of batch size that raises tokens per second lowers this number in direct proportion. That is why the scheduler is the most important cost component in the system.

Key idea. The KV cache caps the batch, the batch sets throughput, and throughput sets the GPU bill. For prompt-heavy traffic, prefill compute can dominate the fleet.

1. API design

Before reading on. The fleet is overloaded. Should the API queue the request and wait, or reject it at once? What should the client see?

Neither extreme works alone. A short, bounded queue absorbs small bursts. Past a wait budget, the API rejects at once with a clear status and a retry hint. A client that waits 60 seconds and then times out is worse off than one that gets a fast rejection and retries with backoff.

3.1 Generate

POST /v1/messages
Headers: x-api-key, Idempotency-Key (optional, for non-streamed calls)
{
  "model": "model-large-2026-09",
  "messages": [{"role": "user", "content": "..."}],
  "max_tokens": 1024,
  "temperature": 0.7,
  "stream": true
}

stream events (server-sent events):
  message_start   {id, model}
  content_delta   {text}
  message_stop    {stop_reason, usage: {input_tokens, output_tokens}}

non-streamed response:
  {id, content, stop_reason, usage}

3.2 Errors

400  prompt plus max_tokens exceeds the model's context window
401  bad API key
429  rate limit for this key or tier, with Retry-After
529  service overloaded, with Retry-After
500  internal error; safe to retry non-streamed calls with the same Idempotency-Key

The server counts usage and bills from its own count. max_tokens is required, because the scheduler uses it to reserve capacity.

Key idea. Reject fast and clearly under overload, separate per-key limits (429) from fleet overload (529), and require max_tokens so the scheduler can plan.

1. Data model

Most state lives in memory on the serving path. Durable storage holds only configuration and usage.

4.1 Request state on the scheduler

Request {
  id, api_key, tier, model_version,
  prompt_tokens: int[], max_tokens, sampling,
  arrived_at, deadline,           // admission deadline for queued requests
  state: queued | prefilling | decoding | finished | cancelled,
  generated: int,                 // tokens produced so far
  block_table: int[]              // KV blocks owned by this sequence
}

4.2 KV memory on each replica

free_blocks: stack of block ids
block_refcount[block_id]: int     // >1 when sequences share a prefix
prefix_index: hash(prefix tokens) -> block ids

4.3 Fleet registry and usage

replica(id, model_version, gpus, status, queue_len, kv_free_blocks, last_heartbeat)
usage(request_id, api_key, model, input_tokens, output_tokens, finished_at)  // append only
limits(api_key, tier, tokens_per_minute, requests_per_minute)                // config store

The router reads the registry every few hundred milliseconds. Usage records go to a log, then to billing, off the request path.

Key idea. The hot path keeps state in memory: queues, block tables, and replica load. Durable writes happen only for usage, and never block a token.

1. High-level design

The design is easiest to follow when built from the smallest version that works. Each failure pulls in the next component.

5.1 One request per GPU pass

The simplest server loads the model and runs one request at a time: prefill, then decode until done, then the next request.

flowchart LR
  C([Client]):::user --> A[API server]:::svc --> M[Model on GPUs<br/>one request at a time]:::store
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

It works, and it wastes the GPU. Section 2.3 showed one sequence gets 190 tokens per second from hardware that can produce 22,000. Every other request waits in line.

5.2 Fix 1: batch requests together

Collect several requests and run them in one pass. Each weight read now serves every sequence in the batch.

flowchart LR
  C([Clients]):::user --> A[API server]:::svc --> Q[[Queue]]:::svc --> B[Static batcher<br/>wait for N]:::new --> M[Model on GPUs]:::store
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

Throughput jumps. But the batch runs until its longest answer finishes. A 20-token answer waits for a 1,000-token answer beside it, and its slot sits empty for most of the batch. New requests wait for the whole batch to drain.

5.3 Fix 2: schedule every decode step

Replace the static batcher with a scheduler that re-forms the batch at every step. When a sequence finishes, it leaves at once and its slot goes to a waiting request.

flowchart LR
  Q[[Waiting queue]]:::svc --> S[Iteration scheduler<br/>runs every step]:::new
  S -->|batch for this step| M[Model on GPUs]:::store
  M -->|finished sequences leave| S
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

The batch stays full, and a new request starts within one step, about 10 ms. The new limit is memory: the scheduler cannot admit a sequence without room for its KV cache.

5.4 Fix 3: page the KV cache

With contiguous allocation, each request reserves memory for its maximum length up front. Most of that reservation is never used, so few sequences fit. Paged allocation hands out small blocks as each sequence grows. The scheduler admits a request when enough free blocks exist for its prompt and a few steps of growth.

flowchart LR
  S[Iteration scheduler]:::svc --> BM[Block manager<br/>16-token blocks]:::new
  BM --> KV[(KV memory pool)]:::store
  S --> M[Model on GPUs]:::store
  M --> KV
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

When memory runs out mid-flight, the scheduler preempts the newest sequence. It frees that sequence's blocks and puts the request back in the queue to recompute later. Shared prefixes, such as a long system prompt used by every request from one customer, occupy one set of blocks with a reference count.

5.5 Fix 4: many replicas and a load-aware router

One replica cannot carry the traffic, and one replica is a single point of failure. Run many replicas and put a router in front. The router must be load-aware: round-robin sends new work to a replica whose KV memory is already full while another sits half empty.

flowchart LR
  G[Gateway]:::svc --> RT[Router<br/>least queued tokens]:::new
  RT --> R1[Replica 1]:::store
  RT --> R2[Replica 2]:::store
  RT --> R3[Replica N]:::store
  R1 -. load report .-> REG[(Replica registry)]:::new
  R2 -. load report .-> REG
  R3 -. load report .-> REG
  REG -.-> RT
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

Replicas report queue length and free KV blocks every few hundred milliseconds. The router picks the replica with the least queued work for the requested model version. It also prefers a replica that already holds the request's prefix in cache, when one is close to the best choice.

5.6 Fix 5: admission control and tiers

Bursts arrive faster than the fleet can grow. Without admission control, queues grow without limit, every request slows down, and clients time out after they have already used GPU time. Add an admission layer at the gateway.

flowchart LR
  C([Clients]):::user --> GW[Gateway]:::svc
  GW --> RL{Per-key limit}:::new
  RL -->|over| R429[429]:::bad
  RL -->|ok| AD{Estimated wait<br/>under budget?}:::new
  AD -->|no| R529[529]:::bad
  AD -->|yes| TQ[[Paid queue / free queue]]:::new --> RT[Router]:::svc
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

The gateway estimates the wait from queued tokens and recent fleet throughput. If the wait passes the budget, it rejects at once with 529. Paid and free traffic have separate queues, and the router serves paid first.

5.7 The composed design

sequenceDiagram
  autonumber
  actor C as Client
  participant G as Gateway
  participant R as Router
  participant S as Replica scheduler
  participant M as Model (GPUs)
  C->>G: POST /v1/messages (stream)
  G->>G: auth, per-key limit, input safety check
  G->>R: admit (tier, model, prompt tokens)
  R->>S: enqueue on least-loaded replica
  S->>M: prefill (possibly in chunks)
  M-->>S: first token
  S-->>G: token
  G-->>C: content_delta
  loop every decode step
    S->>M: step for the whole batch
    M-->>S: one token per sequence
    S-->>G: tokens
    G-->>C: content_delta
  end
  S-->>G: stop + usage
  G-->>C: message_stop
  G--)G: usage record to log (async)

Each component answers one failure of the simpler design. Batching fixes wasted compute. Step-level scheduling fixes idle slots. Paged memory fixes wasted KV space. The router spreads load. Admission control keeps bursts from turning into timeouts.

Key idea. Build it as a chain of fixes: batch, schedule per step, page the cache, route by load, and admit only what the fleet can serve.

1. Deep dives

Five decisions carry the round: how prefill and decode share GPUs, how the system survives a burst, what happens when a GPU dies, how models roll out, and what to measure.

6.1 Prefill and decode interference

Before reading on. A request with a 50,000-token prompt joins a busy replica. What happens to the other 200 users on that replica, and what would you do about it?

Prefill of 50,000 tokens is a large burst of compute. If it runs as one pass, every decoding sequence on the replica waits for it. Their time per output token spikes from about 11 ms to hundreds of milliseconds for that step, and users see the stream freeze.

Two fixes exist, and they combine.

Chunked prefill splits a long prompt into chunks, such as 512 tokens, and adds one chunk to each decode step. Decode steps get a little slower, but they never stall. The long request's time to first token grows a little, because its prefill is spread over many steps.

Disaggregation runs prefill and decode on separate replica pools. A prefill replica processes the prompt, then sends the KV cache to a decode replica over a fast network. Each pool is tuned for its own phase: prefill pools for compute, decode pools for memory. Section 2.4 showed prefill can need more GPUs than decode, so sizing the pools separately saves money. The cost is moving the KV cache: 50,000 tokens × 0.33 MB is about 16 GB for that request, which needs a fast interconnect.

flowchart LR
  RT[Router]:::svc --> PF[Prefill pool<br/>compute-heavy]:::store
  PF -->|KV cache transfer| DC[Decode pool<br/>memory-heavy]:::store
  DC -->|tokens| RT
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

Start with chunked prefill, because it is simple and fixes the stall. Move to disaggregation when long prompts are common and the fleet is large enough to split.

6.2 Surviving a burst

Before reading on. Traffic doubles in five minutes. A new GPU replica takes eight minutes to start. What protects users during the gap?

The autoscaler cannot absorb a burst that is shorter than its start time. A replica needs a machine, a container, and hundreds of gigabytes of weights loaded to GPU memory. So the system needs three layers of defense.

Admission control holds the line. The gateway estimates each request's wait from queued tokens divided by recent throughput. If the estimate passes the budget, it returns 529 at once. Clients back off with jitter, and the queue stays short enough that admitted requests meet their targets.

Tiered shedding decides who waits. As the queue grows, the gateway tightens free-tier limits first, then rejects free traffic entirely. Paid traffic keeps a reserved share of capacity. The limits follow observed queue depth. A static token bucket per key limits one client, but it does nothing when the whole fleet is short.

Faster capacity shortens the gap. Keep a warm pool of replicas with weights already loaded and no traffic. Cache weights on local NVMe disks. Pull weights from peer machines at full network speed, which is its own design problem (see model weight distribution).

Scale on the right signal. GPU utilization looks high on a replica that could take more work, because decode keeps the GPU busy even at small batch sizes. Scale on queued tokens and free KV memory.

flowchart LR
  B[Burst]:::bad --> A{Admission:<br/>wait under budget?}:::svc
  A -->|no| F[529 + Retry-After<br/>free tier first]:::bad
  A -->|yes| Q[[Queue]]:::svc --> RP[Replicas]:::store
  QD[Queued tokens<br/>KV free]:::svc --> AS[Autoscaler]:::svc --> WP[Warm pool<br/>weights loaded]:::store --> RP
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

6.3 A GPU fails mid-batch

Before reading on. One GPU in an 8-GPU replica fails. What happens to the 250 sequences running on that replica?

With tensor parallelism, each GPU holds a slice of every layer. Losing one GPU stops the whole replica, and every sequence in its batch loses its KV cache. The router marks the replica unhealthy after missed heartbeats or errors, stops sending it work, and hands its in-flight requests back to the gateway.

What happens next depends on the request.

• Non-streamed requests are simple. Nothing reached the client yet, so the gateway resubmits to another replica. The client's idempotency key means a client-side retry does not create a second billed request.
• Streamed requests already sent tokens. One option is to resubmit the prompt plus the tokens already sent, and continue. The new replica prefills them and generates the rest. The answer stays coherent, though with sampling on, its style can shift at the join. The other option is to end the stream with a retryable error and let the client decide. Many APIs choose the error for simplicity. Say which one you pick and why.

Usage is recorded for tokens actually delivered. Nobody pays for the lost work.

6.4 Several models and safe rollouts

Before reading on. A new model version is ready. How do you move traffic to it without risking the whole fleet?

Each replica serves exactly one model version. The registry maps each version to its replicas, and the router only sends a request to replicas of the version it asks for. Model aliases, such as model-large-latest, resolve to a version at the gateway.

A rollout is a traffic shift. Start a small pool of new-version replicas. Send a small share of alias traffic to them. Compare error rates, latency, output checks, and evaluation scores against the old version. Grow the share in steps, and keep the old pool warm until the new one proves itself. Rollback is a routing change, which takes seconds.

Rarely used models waste GPUs when they sit loaded. Keep them on a small shared pool, accept a cold start of minutes for the first request, and tell clients through a clear error or longer wait.

6.5 What to measure

Track TTFT and TPOT at p50, p95, and p99, split by tier and model. Track queue wait, admission rejections, batch size, free KV memory, preemptions, tokens per second per replica, and cost per million tokens. A rise in preemptions means the scheduler admits more than memory can hold. A drop in batch size with a full queue means something limits admission, often fragmentation or a stuck replica.

Key idea. Measure what the scheduler controls: batch size, KV memory, preemptions, and queue wait, alongside the two latency numbers users feel.

6.6 Prefix caching and multi-turn chat

Before reading on. A chat user sends turn 12 of a conversation. The prompt is 11 earlier turns plus the new message, 9,000 tokens in all. Only the last 300 tokens are new. What does the server recompute, and can it avoid that?

Without prefix caching, the server prefills all 9,000 tokens again, although it computed the KV cache for the first 8,700 of them one turn ago. For a chat workload, most prefill work is repeated work.

Prefix caching keeps KV blocks after a request finishes, indexed by a hash of the token blocks they cover. When a new request arrives, the scheduler walks its prompt in block-sized steps, hashing each prefix, and reuses every block already in memory. Only the new suffix needs prefill.

For practice, take the numbers from section 2.4: prefill costs about 2 × 70 billion operations per token. Recomputing 8,700 tokens costs about 1.2 × 10^15 operations; at an effective 500 teraflops per GPU across 8 GPUs, that is about 0.3 seconds of a replica's full compute, spent on every turn. With the prefix cached, the new 300 tokens take about 10 ms. Time to first token for turn 12 drops from a few hundred milliseconds to tens of milliseconds, and the replica serves many more turns per second.

The same mechanism serves shared system prompts. If every request from a customer starts with the same 3,000-token system prompt, one copy of its blocks serves all of them, with a reference count per block.

Three details make it work in production:

• Routing must follow the cache. The cached blocks sit on one replica. The router hashes the conversation or the prefix and prefers the replica that holds it, unless that replica is overloaded. This is consistent hashing with a load cap.
• Eviction must be smart. Cached blocks compete with running sequences for memory. Evict least recently used blocks first, and never evict blocks a running sequence is using.
• Correctness depends on exact tokens. A prefix matches only if every token is identical, including the system prompt and any tool definitions. Put stable content first in the prompt and variable content last, so prefixes line up.

flowchart LR
  R[Request: 8,700 old + 300 new tokens]:::svc --> H[Hash prompt in 16-token blocks]:::svc
  H --> M{Blocks in cache?}:::svc
  M -->|first 8,700 tokens: yes| REUSE[Reuse KV blocks]:::store
  M -->|last 300: no| PF[Prefill 300 tokens]:::store
  REUSE --> D[Decode]:::svc
  PF --> D
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

6.7 Rate limits that follow capacity

Before reading on. A customer's plan allows 400,000 tokens per minute. How do you enforce that across 20 gateway servers, and what happens when the fleet loses half its GPUs?

Enforce per-key limits in tokens, not requests: a request with a 100,000-token prompt costs far more than one with 100 tokens. Charge the prompt tokens plus max_tokens at admission, then refund the unused output tokens when the request finishes. That keeps a client from sending many small-looking requests that each generate thousands of tokens.

Across 20 gateways, keep the counters in a shared in-memory store with atomic increments, one key per API key per minute window, and check with a sliding window counter. Each check adds one round trip of well under a millisecond. For very high-volume keys, let each gateway lease a slice of the budget locally and refill from the central counter when it runs out, which removes the round trip at the cost of small overshoot.

Per-key limits protect the fleet from one client. They do not protect the fleet from itself. When half the GPUs fail, total capacity halves while every customer is still within their plan. So add a second, dynamic layer: a fleet-wide admission budget derived from measured throughput. The gateway compares queued tokens with what the fleet can serve within the wait budget, and scales the effective limit per tier. Free tier drops first, then paid limits shrink proportionally, and every rejection returns 529 with a Retry-After based on the queue estimate.

This is the answer interviewers push for: static token buckets per key, plus dynamic throttling tied to observed queue depth.

1. Variants

7.1 The fixed-batch variant

Some interviewers fix the GPU contract and rule out continuous batching, autoscaling, and rate limits. You get a call you cannot change, and you design only the dispatch layer around it. For practice:

run_batch(inputs: list[str]) -> list[str]
  - 1 to 50 inputs per call
  - returns one output per input, in order
  - always takes 200 ms, whatever the batch size
  - each GPU runs one call at a time
traffic: 2,000 requests per second, synchronous HTTP, p95 budget 500 ms

Capacity. A full batch serves 50 / 0.2 = 250 requests per second per GPU, so 8 GPUs is the floor. At half-full batches the same traffic needs 16. The batcher decides which bill you pay.

The batcher. Flush a batch when the queue holds 50 requests or when the oldest request has waited T milliseconds, whichever comes first. Dispatch to any idle GPU.

pending: queue per tier (paid first)
inflight: batch_id -> (gpu, request_ids, attempt)

loop:
  wait until len(pending) >= 50 or oldest.wait >= T
  batch = take up to 50, paid before free
  gpu = next idle worker
  inflight[batch_id] = (gpu, ids, attempt=1)
  outputs = gpu.run_batch(batch)
  for each (request, output) in order: resolve the waiting HTTP call
  delete inflight[batch_id]

on worker failure:
  for each batch on that worker: re-dispatch to another worker, attempt += 1

Choosing T. The worst case for one request is T, plus the time waiting for a free GPU, plus 200 ms. To keep p95 under 500 ms with room for queueing, T should be about 150 ms or less. At 2,000 requests per second, 50 requests arrive every 25 ms, so batches fill by size long before T. At 20 requests per second, batches never fill, and T sets the latency.

Free and paid. Two queues feed the same batcher. Each batch takes paid requests first, then fills its remaining slots with free ones. Under overload, free requests wait longest and time out first.

7.2 The design-doc delivery

The same system often arrives as a document to critique: an inference server with planted weaknesses. The interviewer mostly cares about the batching strategy, so get to it fast. The method is in the design-doc review solution.

7.3 Very long contexts

At 200,000 tokens, one sequence needs about 65 GB of KV cache in this model. It crowds out dozens of normal requests. Route very long requests to a separate pool, use chunked prefill always, and consider a lower-precision KV cache to halve its size, with an evaluation to confirm quality holds.

7.4 Several regions

Run a full fleet per region, and route users to the nearest healthy region. When one region is short on GPUs, the gateway can send overflow to another region for non-interactive traffic. Interactive traffic stays local, because a cross-region hop adds latency to every token.

7.5 At ten times the traffic

At 5,000 requests per second, section 2.4's math gives roughly 3,500 GPUs before headroom. Four things change.

• Disaggregation becomes worth it. Prefill needs about 60% of the fleet in that estimate; sizing a separate prefill pool with compute-optimized settings and a decode pool with memory-optimized batch sizes saves a meaningful share of GPUs.
• Routing becomes hierarchical. One router cannot track 450 replicas' queues in real time. Split the fleet into cells of 30 to 50 replicas, each with its own router, and put a thin global layer in front that picks a cell by load and prefix affinity.
• Weight distribution becomes a system. Adding 50 replicas during a burst means moving tens of terabytes of weights quickly; peer-to-peer distribution is required, not optional.
• Capacity planning moves to forecasts. GPUs are procured weeks or months ahead. Admission control handles minutes; forecasts of weekly peaks handle the fleet size.

1. The transferable pattern

An inference server is a scheduler over a scarce, stateful resource. The resource is GPU memory, and the state is each sequence's KV cache. The same shape appears wherever expensive hardware serves many small, variable jobs: GPU training clusters, video transcoding, even database connection pools. The questions transfer: what is the unit of scheduling, what caps concurrency, what does the scheduler do when the cap is hit, and how does admission keep queues short?

Review: the 30-second answer

• Decode is memory-bound, so batch. One weight read serves every sequence in the batch.
• Schedule at every step. Continuous batching keeps the batch full and starts new requests within one step.
• Page the KV cache. Blocks on demand fit far more sequences and let prefixes share memory.
• Route by load and admit by wait. Queued tokens and free KV memory drive routing, autoscaling, and 529s, with free traffic shed first.
• Separate prefill from decode when prompts are long. Chunk first; disaggregate when the fleet is large.

Quiz

Sources and further reading

• Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI 2022) introduced iteration-level scheduling, the basis of continuous batching.
• Efficient Memory Management for Large Language Model Serving with PagedAttention describes paged KV memory and prefix sharing.
• Sarathi-Serve: Taming Throughput-Latency Tradeoff in LLM Inference covers chunked prefill and stall-free scheduling.
• DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving makes the case for separate prefill and decode pools.
• NVIDIA H100 specifications give the memory bandwidth and compute figures used in the estimates.
• Cost and latency covers the same tradeoffs from the customer's side.