AI Evaluations for FDE Interviews and Production Systems

If you learn one thing in the Production section, learn this one. Evaluations — evals — are how you know a probabilistic system works, and the habit of building them first is the single clearest line between an engineer who ships demos and one a customer trusts in production. It is the core of the role's technical signal.

Why this is the job, not a chore

Here is the pattern every FDE engagement hits. The model performs well in a demo. You run it against the customer's real data — decades of inconsistent records, edge cases no one documented, questions a regulator has to be able to sign off on — and it falls apart. Closing that gap is the deployment, and you cannot close a gap you can't measure.

Before reading on: a customer asks, "How do I know this assistant is accurate?" You could say "we tested it and it looked good." Why is that answer disqualifying, and what would a senior FDE say instead?

"It looked good" is disqualifying because it doesn't survive the next question: accurate on what, measured how, and how would you know if it got worse tomorrow? The senior answer names a number against a dataset: "We grade it on a fixed set of representative questions, scored by domain experts and an automated judge; it's at a known accuracy level, and we re-run the suite daily so regressions surface in a day." That one sentence carries the dataset, the graders, the number, and the schedule that keeps it honest.

Three layers, built in order

Evals are not one thing. You build them in layers, each answering a question the last one couldn't.

1. Offline evals — does it work at all?

Start with a fixed dataset of inputs paired with a notion of the right answer, and run it before anything ships. This is your regression suite. For a retrieval assistant, that's a set of real questions with the passages or answers they should surface. The point of offline evals is repeatability: the same suite, run on every change, so you can tell whether a new prompt or model helped or hurt. Without it, every "improvement" is a guess.

2. Online evals — does it work in the wild?

Offline data is never the full distribution of what real users do. Online evals measure the live system: sampling real traffic, logging outputs, tracking thumbs-up/down, escalations, and task completion. The gap between your offline score and your online score is itself a signal. A wide gap means your test set doesn't look like reality, and your test set is what needs fixing.

3. LLM-as-judge — how do you score "good" at scale?

Most FDE outputs aren't right or wrong; they're better or worse — a summary, an answer, a draft email. You can't human-grade every one. The standard move is to use a strong model as a judge: give it the input, the output, and a rubric, and have it score. It's cheap and scales, with two failure modes to respect: judges are biased (toward longer answers, toward their own style) and they drift. So you calibrate the judge against human grades on a sample, and you keep humans in the loop for the high-stakes slice. A judge scales human judgment rather than replacing it.

Calibration is a procedure, and it is short enough to recite: sample real outputs from the system, have humans — ideally the customer's domain experts — grade them against the same rubric the judge uses, and measure how often the judge agrees with the humans. When they diverge, the disagreements are the diagnostic: an ambiguous rubric gets rewritten, a biased judge gets a different prompt or model, and if the humans disagree with each other, the rubric was underspecified for people too. The benchmark to know: Zheng et al., studying this systematically, found that strong LLM judges like GPT-4 can "achiev[e] over 80% agreement, the same level of agreement between humans" — which makes human-to-human agreement the natural bar, since a judge near it scores about as consistently as your experts do. The same paper documents the judge biases you are calibrating against: position bias, verbosity bias, and self-enhancement bias.

The discipline that earns trust: run it daily

A one-time pre-launch eval goes stale the moment the corpus or the model changes. In a regulated setting, "it was accurate at launch" is not a defense. The move that actually earns adoption is a standing regression suite that runs on every change and on a schedule, so drift surfaces in a day rather than in a customer meeting. The Morgan Stanley engagement is the worked example: the team ran daily regression testing against a fixed question set, and advisor adoption reached near-universal levels. The sources document both facts side by side; reading the first as what earned the second is this course's interpretation, argued on the engagement page. Trust isn't a launch event; it's a process the customer can see is always on.

Bad / Good / Great — "how do you evaluate this system?"

Bad — "I tried a bunch of prompts and the outputs looked right." Anecdote, not measurement. It can't detect a regression, can't be defended to a regulator, and tells the interviewer you've never run something real.

Good — "I built a test set of representative cases and scored accuracy before launch." A real offline eval, and solid mid-level work. The gap to Great: it's a snapshot, it doesn't account for the live distribution, and it says nothing about how you'd catch the system getting worse.

Great — "Offline regression suite on every change, online metrics on live traffic, an LLM-judge calibrated against expert grades for the subjective slice, and a daily run so drift surfaces fast. The customer's compliance bar is encoded in the judge's rubric." Layered, continuous, and tied to the customer's actual definition of good — the structure a real regulated deployment used.

Is the change real?

A standing suite produces a stream of scores, and a stream of scores invites a question the layers above cannot answer. The suite read eighty-two percent yesterday; you changed the prompt; today it reads eighty-four. Improvement or noise? An eval score on a finite case set is an estimate of how the system would perform on the full distribution, and every estimate carries sampling noise. The smaller the set, the wider the noise band — and a delta smaller than the band is a coin flip wearing a percentage.

The width of the band comes from the standard error of a proportion, and the arithmetic fits in a few lines:

Standard error of a proportion:   SE = sqrt(p × (1 − p) / n)

At a score of p = 0.8 on n = 200 cases:
    0.8 × 0.2 = 0.16
    0.16 / 200 = 0.0008
    sqrt(0.0008) ≈ 0.028          → one SE ≈ 2.8 points

So an 82% → 84% move is a two-point delta, well inside one SE.

Growing the set shrinks the band:
    0.16 / 1000 = 0.00016
    sqrt(0.00016) ≈ 0.013         → at n = 1,000, one SE ≈ 1.3 points

Three operating rules fall out of that arithmetic. First, size the eval set to the delta you need to detect: small deltas need big sets, and if the set you can afford is small, only large movements are readable — decide which deltas matter when you build the suite. Second, prefer a paired comparison on the same cases: run the old and new versions on the identical suite, set aside every case where the two agree, and read the result from the cases where they disagree. If the new version wins most of the disagreements, the change is doing something; if the disagreements split about evenly, you are looking at noise. The shared cases cancel out of the comparison, which lets it resolve smaller differences than two independently sampled scores can. Third, when a release gate rides on the number, state the noise band next to the score — "eighty-four, plus or minus three" tells the person signing off exactly how much weight the number can bear.

The judge-calibration procedure above runs on the same logic: an agreement rate between judge and experts is also a proportion measured on a finite sample, so it carries the same band, and a judge calibrated on a handful of cases is calibrated in name only.

A minimal eval harness

You don't need a framework to start. The skeleton below is the whole idea: a dataset, a scorer, an aggregate. (Illustrative code.)

# eval_harness.py — the smallest thing that is a real eval
import json

def load_cases(path):
    # each case: {"input": ..., "expected": ...(optional), "rubric": ...}
    return [json.loads(line) for line in open(path)]

def score(case, output, judge):
    """Return a 0..1 score. Swap in exact-match, retrieval-hit, or LLM-judge."""
    if "expected" in case:
        return float(case["expected"].lower() in output.lower())  # cheap offline check
    return judge(case["input"], output, case["rubric"])           # subjective -> judge

def run(cases, system, judge):
    results = [(c, system(c["input"])) for c in cases]
    scores = [score(c, out, judge) for c, out in results]
    return {
        "n": len(scores),
        "mean": sum(scores) / len(scores),
        "fails": [c["input"] for (c, _), s in zip(results, scores) if s < 0.5],
    }

# Run it on every change. Check `mean` against last run; investigate `fails`.

Three properties make it a real eval and not a demo: it's repeatable (same cases every run), it aggregates (a number you can track over time), and it surfaces failures (the list you go fix). Everything else — better datasets, a calibrated judge, online sampling — is refinement on this spine.

What to carry into the interview

In the AI system design round, when the prompt has any accuracy, safety, or compliance bar, do not open with the model or the vector store. Open with how you'll measure correctness, continuously, against a dataset that looks like the customer's reality, and where the human stays in the loop. Sketching the architecture proves little; the eval strategy is where the answer is decided.

Next: Guardrails & Safety — the eval measures quality; guardrails bound the failures that slip past it.