Retrieval-Augmented Generation and Vector Search for FDEs

A model knows two things: what it learned in training, and what you put in the context this call. It does not know your customer's private documents, and its training is frozen in the past. Retrieval-augmented generation is how you close that gap. You fetch the relevant text at question time and put it in the prompt, so the model answers from the customer's own data instead of from its memory. For an FDE, this is the most common architecture there is, because the customer's value is almost always locked in data the model never saw.

Why this is the job

The pattern every FDE engagement hits: the customer has decades of documents, and they want answers grounded in those, private and current. A model alone will confidently make something up (it has no source); RAG gives it sources. The Morgan Stanley engagement is the cited example: an assistant over the firm's own research library, where grounding answers in vetted internal research was the entire point. Get retrieval right and the system is trustworthy; get it wrong and no model quality saves you.

Before reading on: your RAG assistant gives a confident, wrong answer. The model is fine and the prompt is fine. Where do you look first, and it isn't the model?

You look at retrieval. If the passages you fed the model were irrelevant or missing the answer, the model had nothing true to work from, so it filled the gap with fiction. In RAG, retrieval quality is the ceiling on answer quality. The model can only be as right as the context you retrieved.

How it works, in five steps

• Chunk. Split the corpus into passages small enough to be precise and large enough to be self-contained. Chunking strategy quietly determines retrieval quality.
• Embed. Turn each chunk into a vector with an embedding model, so semantic similarity becomes geometric closeness.
• Index. Store the vectors in a vector database with an approximate-nearest-neighbor index, so lookup over millions of chunks is fast.
• Retrieve. At query time, embed the user's question and pull back the top handful of nearest chunks by meaning.
• Generate. Splice those chunks into the prompt as grounding context and have the model answer from them, ideally citing which chunks it used.

A worked trace — and how you'd measure it

Here is the loop above run once, for an assistant over a firm's research library. (Illustrative: the documents and scores are invented; study the shape.)

Question: "What's our current outlook on European industrials?"

Top 3 retrieved chunks (by vector similarity):
[1] eu-industrials-2026-05.pdf, p.2   (score 0.83)
    "We upgrade European industrials to overweight, driven by defense
     spending and grid investment…"
[2] global-sectors-2026-05.pdf, p.11  (score 0.79)
    "Within global industrials we prefer Europe over the US on valuation…"
[3] eu-macro-2026-04.pdf, p.7         (score 0.74)
    "Euro-area surveys point to a shallow manufacturing recovery through
     the second half…"

Grounded answer:
"The house view is overweight European industrials, upgraded in May on
defense spending and grid investment [1]; within global industrials the
preference is Europe over the US on valuation [2], against a shallow
manufacturing recovery [3]."

Every claim in the answer traces to a numbered chunk, which is what lets a user, or a groundedness guardrail, verify it.

The trace also shows exactly what to measure. Build a golden set: real questions paired with the passages that answer them, judged by someone who knows the corpus. Then ask two questions of the retriever. First, recall at k: for what fraction of questions does a right passage appear in the top k results you actually put in the prompt? Recall is the classic information-retrieval measure, "the fraction of relevant documents that are retrieved" (Manning, Raghavan & Schütze, Introduction to Information Retrieval, ch. 8), and the "at k" cutoff convention scores it at the cutoff that matches your context budget. Second, mean reciprocal rank (summary): the average, across questions, of one divided by the rank of the first relevant passage. It comes from the Text REtrieval Conference (TREC) question-answering evaluations (Voorhees, 1999) and rewards putting the right passage first, where it most influences the answer. On how large the golden set should be, no vendor publishes an authoritative figure; Anthropic's guidance is to "Prioritize volume over quality: More questions with slightly lower signal automated grading is better than fewer questions with high-quality human hand-graded evals". When the Great answer below says to evaluate retrieval directly, this is the machinery it means.

The tradeoffs an interviewer probes

• Retrieval quality is everything. Better chunking, better embeddings, re-ranking the candidates, and hybrid keyword-plus-vector search all move the ceiling. Most "the model is bad" complaints are retrieval failures in disguise.
• More context is not free. Every retrieved chunk lengthens the prompt, which raises both cost and latency (Cost & Latency). Retrieve few, highly relevant chunks. Precision beats volume, and it's cheaper.
• Groundedness is a safety property. The whole promise is answering from the sources, so a guardrail that checks the answer is actually supported by the retrieved passages, plus citations a user can verify, is what makes RAG trustworthy in a regulated setting.
• Freshness is retrieval, not retraining. New data just gets re-indexed instead of retrained into the model, which is why RAG is the practical way to keep an assistant current.
• Retrieval must respect who's asking. In a customer's corpus, not every user is cleared for every document, so retrieval has to filter by the requester's permissions before it ranks. Skip that and the assistant becomes a way to read files you were never granted, since it will ground an answer in whatever chunk it retrieved, including one the asker could never open. Enforce access at the index as a pre-filter (metadata filters or per-tenant partitions), not a post-filter on the results, so an unreadable document is never a candidate and you don't spend the retrieval budget on chunks you discard; the tradeoff is that tight scoping shrinks the candidate pool and can cost recall for the users who can see least. It is the security-and-compliance boundary meeting retrieval, and generic RAG tutorials almost never mention it.

Bad / Good / Great — "how do you make it answer from our documents?"

Bad — "I'd fine-tune the model on their documents." As a way to get facts into the model, this is the wrong tool: expensive, slow to update (new documents mean a new training run), and the result still can't quote a source. Fine-tuning has legitimate jobs, like enforcing an output format, teaching domain-specific behavior and tone, or distilling a task onto a smaller, cheaper model, and real deployments pair it with retrieval. The Morgan Stanley team shaped the assistant's behavior through advisor grading and retrieval tuning, while every answer stayed grounded in retrieved research. What makes this answer Bad is reaching for a retrain to get facts in, when retrieval is the standard tool for that job.

Good — "embed the documents, retrieve the top matches for a question, and put them in the prompt." The correct architecture. Solid. The gap: no sense that retrieval quality is the ceiling, and nothing on grounding, cost, or how you'd know it's wrong.

Great — "RAG: chunk and embed the corpus into a vector store, retrieve a few high-relevance passages per query with re-ranking, and answer from them with citations. I'd treat retrieval quality as the ceiling and evaluate it directly (is the right passage in the top results?), add a groundedness check so the answer can't drift from its sources, and keep chunks few to control the prefill cost." You named retrieval as the thing to measure, tied it to grounding and cost, and made it evaluable.

What to carry into the interview

When a prompt involves a customer's own data, RAG is almost always the answer, but the signal isn't naming it. It's knowing that retrieval is the ceiling. Lead with how you'd evaluate retrieval quality, how you'd keep answers grounded in and citing their sources, and how you'd control the token cost of context. Fine-tuning is the wrong reflex for getting facts in.

Next: Tool Use & Function Calling. Retrieval gets data into the model; tools let the model act.