Serving, Latency & Cost

A model that looks good in a notebook can still be unusable in production. Users feel latency, finance feels token cost, and infrastructure feels bursty traffic. Serving is the engineering layer that turns inference into a dependable product endpoint.

For LLMs, performance is mostly about tokens: how many input tokens must be processed, how many output tokens must be generated, and how efficiently requests share the hardware.

The mental model

LLM latency has two major phases:

1. Prefill — process the prompt tokens and build the KV cache.
2. Decode — generate output tokens one at a time, reusing that cache.

def estimate_latency(input_tokens, output_tokens, prefill_tps, decode_tps, overhead_ms=80):
    prefill_ms = (input_tokens / prefill_tps) * 1000
    decode_ms = (output_tokens / decode_tps) * 1000
    return overhead_ms + prefill_ms + decode_ms

print(estimate_latency(2_000, 300, prefill_tps=40_000, decode_tps=120))

The numbers vary by model, hardware, batch size, and provider, but the shape is stable: long prompts hurt prefill; long answers hurt decode.

Prefill, decode, and the KV cache

During prefill, the model reads the whole prompt and stores attention keys and values in a KV cache. During decode, each new token can attend to previous tokens through that cache instead of recomputing the entire prompt.

The cache is why conversation history is expensive twice: it consumes context window and memory on the serving side. More concurrent long-context requests mean less room for batching and more pressure on GPU memory.

Continuous batching

GPUs want batches. Users want low latency. Continuous batching reconciles the two by adding and removing requests from a live batch as tokens are decoded.

Without it, a short request can sit behind a long one. With it, the server keeps the GPU busy while still streaming tokens back to each user.

The trade-off is scheduling complexity: batching improves throughput, but too much waiting to form a batch increases time-to-first-token.

Streaming responses

Streaming does not necessarily reduce total generation time. It reduces perceived latency by sending the first tokens as soon as decode begins.

Use streaming for chat, writing, and support answers where partial text is useful. Avoid it for tiny JSON responses, background jobs, or flows where the caller needs the complete object before doing anything.

Quantization

Quantization stores weights in fewer bits, such as 8-bit or 4-bit instead of 16-bit. It reduces memory use and can improve throughput, especially when memory bandwidth is the bottleneck.

The cost is possible quality loss and hardware-specific behavior. Quantize only after evaluating the real task, not just a generic benchmark.

Caching

The cheapest inference is the one you skip.

• Exact cache — same normalized prompt, same answer.
• Semantic cache — embed the request and reuse answers for near-duplicates.
• Prefix cache — reuse KV state for shared prompt prefixes when the serving stack supports it.
• Retrieval cache — cache search results or document chunks separately from the generated answer.

Cache only when the answer is safe to reuse for that user, time, and permission set.

Pattern recognition

Variations

Worked problems

def latency_budget(trace):
    total = sum(trace.values())
    return {name: seconds / total for name, seconds in trace.items()}

print(latency_budget({"retrieval": 2.0, "prefill": 1.0, "decode": 4.0}))

def serving_cost(decode_tps, output_tokens, hourly_cost, utilization):
    responses_per_second = (decode_tps * utilization) / output_tokens
    responses_per_hour = responses_per_second * 3600
    cost_per_1000 = hourly_cost / responses_per_hour * 1000
    return responses_per_second, cost_per_1000

rps, cost = serving_cost(2_400, 300, 3.60, 0.70)
print(round(rps, 2), round(cost, 4))

def serving_mode(endpoint: str) -> str:
    if endpoint == "chat_assistant":
        return "stream"
    if endpoint == "nightly_ticket_summary":
        return "batch"
    if endpoint == "invoice_json_extraction":
        return "non_streaming_request_response"
    raise ValueError(endpoint)

The classic pitfall