Evaluation & Guardrails

The difference between a demo and a product is everything that happens after the model returns a plausible-looking answer. Because LLM outputs are probabilistic, you cannot ship them the way you ship a pure function — you have to measure quality, constrain behavior, and control cost and latency. This lesson is the operations layer of applied AI.

You cannot improve what you cannot measure

The first instinct — “I read a few outputs and they look good” — does not scale and does not survive a prompt change. You need evals: repeatable tests for a probabilistic system.

• Offline evals run a fixed dataset of inputs (a “golden set”) through your system and score the outputs. Scoring can be exact-match for structured tasks, reference-based metrics, or an LLM-as-judge that rates answers against a rubric.
• Online evals measure real traffic: thumbs up/down, task completion, escalation rate, and A/B tests between prompt or model versions.

Treat prompts and model choices like code: every change should run against your eval set before it ships. A prompt tweak that helps one case often quietly breaks three others — evals are how you catch that.

Guardrails: constraining an unreliable core

Guardrails are the checks around the model that keep bad outputs from reaching users or downstream systems:

• Input validation — filter prompt-injection attempts and off-topic or disallowed requests before they reach the model.
• Output validation — enforce structure (e.g. the response must be valid JSON matching a schema), and check for policy violations or leaked secrets.
• Grounding checks — for RAG, verify the answer is actually supported by the retrieved context and isn’t a hallucination.
• Human-in-the-loop — route low-confidence or high-stakes cases to a person.

A robust pattern is to ask the model for structured output and validate it against a schema, retrying on failure:

from pydantic import BaseModel

class Verdict(BaseModel):
    answer: str
    confidence: float
    sources: list[str]

def safe_generate(prompt, retries=2):
    for _ in range(retries + 1):
        raw = llm(prompt)
        try:
            return Verdict.model_validate_json(raw)  # reject malformed output
        except ValueError:
            continue
    raise ValueError("model failed to produce valid output")

The cost and latency triangle

Every production AI decision trades off three things: quality, latency, and cost. Pushing any one usually moves the others. Your levers:

• Model tiering — use a small, cheap, fast model for easy requests and escalate only hard ones to a larger model. This “router” pattern often cuts cost by an order of magnitude.
• Caching — identical or near-identical prompts can return a cached answer. Semantic caching (via embeddings) catches paraphrases too.
• Streaming — stream tokens to the user so perceived latency drops even when total generation time doesn’t.
• Prompt budgeting — shorter prompts and tighter max_tokens cut both cost and latency; retrieval helps by sending only the relevant context.

The senior mindset

Junior work stops when the model returns something reasonable. Senior work asks: How do I know it’s good? What happens when it’s wrong? What does this cost at a million requests a day, and how fast is it at p99? Answering those questions — with evals, guardrails, and cost controls — is the core of the role.

Pattern recognition

Variations

Worked problems

from dataclasses import dataclass

@dataclass(frozen=True)
class EvalCase:
    case_id: str
    question: str
    required_sources: tuple[str, ...]
    rubric: str
    intent: str
    risk: str

def pass_rate(judgments):
    passed = [j for j in judgments if j["grounded"] and j["policy_ok"]]
    return len(passed) / len(judgments) if judgments else 0.0

def resolution_rate(conversations):
    resolved = 0
    for c in conversations:
        if c["answered"] and not c["escalated"] and not c["reopened_within_24h"]:
            resolved += 1
    return resolved / len(conversations) if conversations else 0.0

The pitfall to avoid