AI Guardrails and Safety for Forward Deployed Engineers

Evaluations tell you whether the system is right on average. Guardrails are what you do about the times it's wrong, and about the inputs trying to make it wrong. In a real deployment, the model will occasionally produce something confident and unacceptable, and a user or an attacker will occasionally hand it something hostile. Guardrails keep either from becoming a customer's incident.

Why this is the job, not a checkbox

An FDE deploys where the cost of a wrong output is real: a regulated advisor, a customer's production database, a tool that can send email or move money. In that setting a single bad answer stops being a user-experience problem and becomes a liability. The Morgan Stanley engagement turned on exactly this: advisors would not trust an assistant that might be confidently wrong in front of a client, so keeping a human reviewing outputs was load-bearing. Guardrails are how you earn the right to deploy at all.

Before reading on: you're asked to add an AI feature that drafts and sends customer emails. Name the failure that should keep you up at night — and it isn't a grammatically awkward sentence.

The failure that matters is the irreversible action taken on a wrong belief: the model emails the wrong client, or the right client the wrong thing. Everything below is about making sure that particular failure can't happen unattended.

Three places a guardrail lives

Build them at the three points where the system meets the world.

1. Input guardrails — before the model

Validate and sanitize what reaches the model. Strip or flag personally identifiable information (PII) before it's sent or logged; enforce length and format; and defend against prompt injection, untrusted text that tries to become an instruction. Prompt injection is the top entry on the OWASP Top 10 for LLM Applications, and the defense is boundary thinking: content the model reads is data, never commands, and you never give the model reachable authority over anything an injected instruction could abuse.

Injection doesn't only arrive typed by a user. The subtler route is indirect prompt injection: instructions planted in content the system will later read, like a retrieved document, a fetched web page, or an email the assistant summarizes. Greshake et al. showed adversaries can "remotely… exploit LLM-integrated applications by strategically injecting prompts into data likely to be retrieved", and the OWASP entry for 2025 draws the same line: "Indirect prompt injections occur when an LLM accepts input from external sources, such as websites or files." That means a RAG pipeline is an injection surface: your own corpus, or anything that feeds it, can carry an attack to the model. The defense is the same boundary thinking applied to retrieval: treat retrieved text as data and never as instructions, delimit it explicitly in the prompt, strip or flag instruction-like content found inside it, and never let anything in retrieved content trigger a tool call without passing the same gate as user input. If your assistant reads documents and can act, this is the attack to assume.

2. Output guardrails — after the model

Check the output before it reaches the user or a downstream system. Filter unsafe content, verify the output is grounded in the sources you retrieved (not invented), redact any PII that slipped through, and validate structure if a downstream system will parse it. For a grounded assistant, the most valuable output guardrail is a groundedness check: does the answer actually follow from the retrieved passages, or did the model fill a gap with confident fiction?

3. Action guardrails — when the system can do things

The moment the system takes actions, "wrong" stops being a bad sentence and becomes a bad event. Bound what it can do unattended:

• A budget: a hard cap on steps, tool calls, or spend before it must stop and report.
• A checkpoint: human approval required before anything irreversible or high-blast-radius (sending the email, running the migration, moving the money).
• A kill switch: a known, fast way to halt and roll back.

The design question is never "what can it do?" It's "what can it do without a human?" For irreversible actions the answer is nothing yet, until eval data earns the autonomy.

Bad / Good / Great — "how do you keep it safe?"

Bad — "we add a disclaimer that the AI can make mistakes." This pushes the entire safety burden onto the user and does nothing about hostile input or irreversible actions. It's the answer of someone who hasn't operated a real system.

Good — "we filter unsafe outputs and validate inputs." Real input/output guardrails, genuinely mid-level solid. The gap: it says nothing about a system that acts, and nothing about how you'd know a guardrail is failing.

Great — "layered: sanitize and injection-check the input, verify groundedness and redact PII on the output, and for anything that acts, a budget, a human checkpoint on irreversible steps, and a kill switch — with every block logged so I can see what's being caught." You bounded the blast radius at all three points and made the guardrails observable.

A minimal guardrail

Guardrails are ordinary code around the model call. The skeleton is the whole idea. (Illustrative code.)

# guard.py — deterministic checks around a non-deterministic model
def handle(request, model, tools):
    if contains_pii(request.text):
        request = redact_pii(request)              # input guardrail
    if looks_like_injection(request.text):
        return refuse("input flagged")             # refuse hostile input

    output = model(request)

    if not grounded_in(output, request.sources):   # output guardrail
        return refuse("answer not supported by sources")

    action = output.proposed_action
    if action and action.is_irreversible:          # action guardrail
        return await_human_approval(action)        # checkpoint, don't execute
    return output

The properties that make it real: it acts on the input and the output and the action, it refuses a hostile request where a lesser system would attempt one, and it stops before anything irreversible. Everything else, like better PII detection or a stronger injection classifier, refines this spine.

Inside the named checks — where the hard part lives

The skeleton names two functions and hides the difficulty inside them, so open them up. looks_like_injection() starts as cheap pattern rules ("ignore your previous instructions" and its cousins) plus a small trained classifier, and neither will catch a novel attack. Treat it as a tripwire with a real false-negative rate, which is one more reason the action gate downstream must hold even when this check passes.

grounded_in() is the one to be honest about in a design conversation: in practice, a groundedness check is itself a model call. You hand a judge model the retrieved passages and the drafted answer and ask whether the answer is supported. The shape:

grounded_in(), unpacked — the check is itself a model call (illustrative)

Judge prompt:
  You are verifying a drafted answer against source passages.
  <passages> …the retrieved chunks… </passages>
  <answer> …the drafted answer… </answer>
  For each factual claim in the answer, decide whether the passages
  support it. Return JSON:
  {"grounded": true|false, "unsupported_claims": ["…"]}

Refuse or route to a human when grounded is false.

That design has three consequences worth saying out loud. It adds cost: a second model call on every guarded request, though a smaller model than the one that drafted the answer usually suffices (Cost & Latency). It adds latency: the check runs in sequence, after the draft and before the user sees anything. And it has its own error rate: the judge will sometimes pass an ungrounded answer and sometimes block a grounded one, so you evaluate the guardrail the way you evaluate everything else, scoring it against human-labeled examples and tracking what it misses (Evaluations). A guardrail whose error rate you have never measured is a guess standing where a control should be.

What to carry into the interview

In the AI system design round, when a system can take actions, do not describe the happy path. Lead with what happens when the model is wrong: draw the line between what runs automatically and what needs a human, name the budget and the kill switch, and say how you'd know a guardrail is catching things. Designing for the moment the model is wrong is the instinct this round exists to test.

Related: Security, Compliance & Data for the regulated-environment version of these constraints, and Evaluations for measuring the quality guardrails bound.

Next: Observability & Debugging. The eval and the guardrails only help if you can see what the system did in production.