OpenAI Interview: Mine Novel Data from a Huge Unlabeled Corpus

Part of the OpenAI system design question bank. The question is representative of the round. The analysis and solution are this site's own.

Problem statement

The corpus has billions of items and no labels. One goal is to select data that would teach a model something new. The other is to find all images of a specific object, such as forklifts or X-ray images with a certain finding. A domain variant sets the task in medical data: mine high-signal training examples from a large clinical corpus.

The specification is thin on purpose. The interviewer is grading how you turn vague goals into definitions you can measure.

Clarifying questions

• What does "novel" mean for this team? Propose: far from the existing training set, and not a near-copy of anything already chosen. Confirm.
• What does "useful" mean? Good quality, and relevant to a target, such as a capability the model is weak at. Ask what the target is.
• How big is the corpus? For practice: 1 billion images and 5 billion text documents.
• What labeling budget exists? Assume a few thousand human labels per week, with expert labels expensive.
• How is success judged? Ultimately, by the model trained on the selected data. Ask whether downstream evaluations exist.
• Privacy and licensing? Ask. For medical data, de-identification comes before anything else.

What makes this hard

Three problems stack.

There are no labels, so every signal is a proxy. "Far from the training set" is measurable, but junk is also far from everything: corrupted files, random noise, and spam are the most novel items in any corpus.

Scale forbids anything expensive per item. A billion items at a millisecond each is eleven days on one machine. Every stage must be cheap per item or run only on a small subset.

And the result is hard to verify. You can check what you selected, but you cannot easily measure what you missed.

So the driving tension is coverage versus cost. Scanning everything with an accurate model is impossible; cheap scans miss things. The design is a funnel: cheap filters over everything, expensive checks on a shrinking set, and human labels at the narrow end.

flowchart TB
  C[(Corpus: billions)]:::store --> F1[Cheap filters: format, size, language, corruption]:::svc
  F1 --> F2[Dedup: exact hash, near-duplicate]:::svc
  F2 --> F3[Embed everything once]:::svc
  F3 --> F4[Score: novelty x quality x relevance]:::svc
  F4 --> F5[Expensive checks on the top slice]:::svc
  F5 --> H[Human review of samples]:::user
  H --> OUT[(Selected set)]:::store
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

Key idea. Define the goal as measurable scores first, then build a funnel: cheap work on everything, expensive work on few, humans at the end.

Key concepts

Embeddings as a common currency

An embedding model maps each item to a vector; similar items land close together. Once every item has a vector, novelty, deduplication, clustering, and search become vector operations. Image-text models embed images and text into the same space, so a text description can search images.

Near-duplicate detection

Exact duplicates share a content hash. Near-duplicates, such as a resized image or a document with a changed header, need similarity. MinHash estimates the overlap between documents' sets of word sequences, and locality-sensitive hashing groups likely duplicates without comparing all pairs. For images, a cosine similarity threshold on embeddings works.

Nearest-neighbor novelty

For each candidate, find its k nearest neighbors in the training set's embedding index. Large distances mean the candidate is unlike anything the model saw.

Active learning

Train a classifier on a few labels, use it to score everything, and send the cases it is least sure about to humans. Each round of labels goes where it helps the classifier most.

Precision and recall without full labels

Precision is measured by labeling a sample of what was selected. Recall needs an estimate: label a random sample of what was not selected, or plant known positives and count how many come back.

Key idea. Embed once, then reuse the vectors for dedup, novelty, search, and classification. Measure precision by sampling selections and recall by sampling rejections.

1. Requirements

Before reading on. Write a one-sentence definition of the output for each goal, precise enough to measure.

1.1 Functional requirements

• Filter out broken, low-quality, and disallowed items.
• Remove exact and near-duplicates, within the corpus and against the existing training set.
• Score items for novelty against the training set, quality, and relevance to a target.
• Select a diverse set under a size budget.
• Find all images containing a specified object, with a labeled confidence.
• Report precision and estimated recall for every selection.

1.2 Non-functional requirements

• Throughput. Process a billion items in days, not months.
• Reproducibility. A selection can be regenerated from recorded model versions, thresholds, and seeds.
• Privacy. Sensitive data is filtered or de-identified before any person sees it.
• Cost. GPU hours for embedding and scoring stay within budget.

1.3 The constraint versus the property

Measurable selection quality is the property. A selection nobody can evaluate cannot be trusted. Per-item cost at billion scale is the constraint. It forces the funnel shape.

1. Back-of-the-envelope estimation

2.1 Embedding the images

If one GPU embeds 2,000 images per second, 1 billion images take 500,000 GPU-seconds, about 139 GPU-hours. On 100 GPUs, under 2 hours of wall time. Decoding and reading images from storage often becomes the bottleneck before the GPU; plan the data loading to keep GPUs fed.

2.2 Vector storage

1 billion × 512 dimensions × 2 bytes = about 1 TB. With product quantization to 64 bytes per vector, the searchable index is about 64 GB, which fits in memory on a few machines. Keep the full vectors on disk for re-ranking.

2.3 Novelty queries

Each item needs a k-nearest-neighbor query against the training set's index. At 10,000 queries per second per machine, a billion queries take 100,000 seconds on one machine, about 28 hours, or under 3 hours on 10 machines.

2.4 Expensive checks

An object detector at 50 images per second per GPU can only run on a small slice. Checking the top 10 million candidates takes 200,000 GPU-seconds, about 56 GPU-hours. Running it on all billion would take 5,600 GPU-hours. That ratio is why detection runs after cheap retrieval.

Key idea. Embedding everything costs about a hundred GPU-hours; running a detector on everything would cost thousands. Cheap first, expensive on the survivors.

1. API design

The system is a batch pipeline with a query service on top.

POST /v1/selections
  {goal: novelty | object_search, target, budget_items,
   filters: {...}, embedding_model, thresholds}
  -> {selection_id}
GET  /v1/selections/:id
  -> {status, items_selected, precision_estimate, recall_estimate, report_url}

POST /v1/object-search
  {description | example_image_ids[], min_confidence}
  -> {search_id}
GET  /v1/object-search/:id/results?cursor=
  -> [{item_id, score, detector_confidence, label_status}]

POST /v1/labels   {item_id, label, labeler_id, task_id}

1. Data model

item            (item_id, uri, type, size, content_hash, source, license, ingested_at)
quality         (item_id, filter_version, passed, reasons[])
embedding       (item_id, model_version) -> vector (stored in the vector index + disk)
dup_cluster     (cluster_id, item_id, is_representative)
novelty_score   (item_id, index_version, knn_mean_distance)
relevance_score (item_id, classifier_version, score)
selection       (selection_id, config_json, created_at)
selection_item  (selection_id, item_id, rank, reason)
label           (item_id, task_id, label, labeler_id, created_at)

Every score records the model or index version that produced it, so a selection can be reproduced.

1. High-level design

5.1 Sample randomly

Pick a random subset of the corpus. It is cheap and unbiased. It is also mostly duplicates of what the model already knows, plus junk, because that is what most corpora contain.

5.2 Fix 1: filters and deduplication

Remove broken files, tiny images, unreadable text, and disallowed content with cheap rules and small classifiers. Remove exact duplicates by hash, and near-duplicates with MinHash for text and embedding similarity for images. In web-scale corpora, deduplication alone often removes a large share of items.

5.3 Fix 2: embed everything once

Run one embedding pass over the survivors. Store vectors in an ANN index. Every later stage reuses them.

5.4 Fix 3: novelty against the training set

Build an index over the existing training set's embeddings. For each candidate, compute the mean distance to its k nearest training neighbors. High distance means novel.

5.5 Fix 4: combine novelty with quality and relevance

Pure novelty selects junk. Multiply by a quality score from a small classifier trained on labeled good and bad examples, and by a relevance score for the target. Then select for diversity: cluster the high-scoring candidates and sample across clusters so one dense new topic does not take the whole budget.

flowchart LR
  E[Embeddings]:::store --> N[Novelty:<br/>kNN distance to training set]:::new
  E --> Q[Quality classifier]:::new
  E --> R[Relevance to target]:::new
  N --> S[Score = N x Q x R]:::new
  Q --> S
  R --> S
  S --> D[Cluster + sample across clusters]:::new
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

5.6 Fix 5: measure

Label a random sample of the selection for precision. Estimate recall by sampling rejected items, or by planting known positives. Train with and without the selection and compare on downstream evaluations.

5.7 The composed pipeline

flowchart TB
  C[(Corpus)]:::store --> Q1[Quality filters]:::svc --> DD[Exact + near dedup]:::svc
  DD --> EM[Embedding pass on GPUs]:::svc --> IX[(ANN index)]:::store
  T[(Training set embeddings)]:::store --> NV[Novelty scoring]:::svc
  IX --> NV
  IX --> QS[Quality + relevance scoring]:::svc
  NV --> SEL[Diverse selection under budget]:::svc
  QS --> SEL
  SEL --> LB[Label samples: precision, recall]:::user
  LB --> REP[(Selection report)]:::store
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;

Key idea. Filter, dedup, embed once, score novelty times quality times relevance, select for diversity, and measure with samples.

1. Deep dives

6.1 Defining novelty so it does not reward junk

Before reading on. Your novelty score's top 1,000 items are mostly blank images, glitch art, and spam pages. What went wrong, and how do you fix it?

Distance measures unfamiliarity, not value. The most unfamiliar items are the ones no normal data resembles, and those are usually broken or worthless.

Three fixes work together. Run quality filters before novelty, so junk never gets scored. Multiply novelty by a learned quality score, trained on a few thousand labels of good and bad items. And look at density: an item far from the training set but close to many other candidates is part of a real new topic; an isolated item far from everything is often an outlier. Prefer dense new clusters over lone points.

Check with people. Before any selection ships, a reviewer looks at a random sample from each score band, and the thresholds move until the top band is clean.

6.2 Deduplication at a billion items

Exact duplicates: hash the content and group by hash. Cheap and exact.

Near-duplicate text: split each document into word sequences, compute a MinHash signature of, say, 128 values, and use locality-sensitive hashing to bucket documents whose signatures agree in several bands. Only documents in the same bucket are compared. This finds pairs with high overlap without comparing a billion documents with each other.

Near-duplicate images: embed, then use the ANN index to find neighbors above a cosine threshold. Group them and keep one representative, usually the highest resolution.

Deduplicate against the existing training set too. An item that nearly duplicates training data is by definition not novel.

6.3 Finding every image of an object

Before reading on. The team wants every image that contains a forklift. Walk through the pipeline and say where people are involved.

1. Text-to-image search. Embed "a photo of a forklift" with the image-text model and search the index for the top 1 million images. This takes seconds and has modest precision.
2. Detector verification. Run an open-vocabulary object detector on those candidates. It confirms whether a forklift is present and where. Precision rises sharply.
3. Seed labels. Label a few hundred detector results by hand, including hard cases: toy forklifts, drawings, pallet jacks.
4. Train a classifier on embeddings. A small classifier on the stored image vectors is cheap enough to score all billion images.
5. Active learning. Send the images the classifier is least sure about to labelers. Retrain. Repeat until precision and estimated recall meet the target.
6. Final pass. Run the detector on everything the classifier scores above a low threshold, to catch what text search missed.

Searching with example images, not just text, helps when the object is hard to describe. Average the embeddings of a few example images and search with that.

6.4 Measuring precision and recall

Precision: label a random sample of 400 selected items. The share that are correct estimates precision, with a margin of about plus or minus 5 percentage points at 95% confidence.

Recall is harder. Two methods:

• Sample the rejected set. Label a random sample of items not selected, and estimate how many positives were missed. With rare objects, most samples will be negative, so stratify: sample more heavily from items with moderate scores.
• Plant known positives. Before the run, hide a set of labeled positives in the corpus. The share recovered estimates recall.

Report both numbers with their margins on every selection.

6.5 Downstream evaluation

The true test of "useful" is the model. Train two small models, one on the baseline data and one with the selection added, and compare them on evaluations for the target capability. If the selection does not move the evaluation, it was not useful, however novel.

6.6 The medical variant

De-identify before anything else: strip names, dates, record numbers, and faces, and keep the raw data in a restricted zone that only the pipeline can read. Use domain-specific encoders, which beat general ones on clinical text and images. Label noise is higher and expert labels cost much more, so active learning matters more: spend expert time only on uncertain cases. Track the source of each item for licensing and consent.

6.7 Running the pipeline at scale

Before reading on. The embedding job reads a billion images from object storage. Where will it bottleneck first, and how do you keep GPUs busy?

Usually on data loading. Each image must be fetched, decoded, and resized before the GPU sees it. If one GPU embeds 2,000 images per second, the loaders must decode 2,000 images per second for it, which can take many CPU cores.

• Shard the input. Split the corpus into shards of, say, 100,000 items each, listed in a manifest. Workers take shards from a queue with leases, so a failed worker's shard is retried.
• Pack small files. Millions of small objects are slow to list and fetch. Pack them into large archive files per shard once, and stream them.
• Decode on CPUs, embed on GPUs. Run enough decode workers per GPU to keep it busy, and measure GPU utilization. Below 80%, add decoders.
• Write results per shard. Each shard's output is one file of item IDs and vectors, written atomically. A rerun of a shard overwrites its file, so retries are idempotent.
• Checkpoint progress in the manifest. The job can stop and resume at any time.

6.8 Bias in the selection

Selection by novelty and a learned quality score can quietly skew the data. A quality classifier trained on a few thousand labels may rate certain languages, dialects, or image styles as low quality because they were rare in its training labels. Novelty can over-select one strange cluster.

Measure the composition of the selected set against the corpus on the attributes you care about: language, source, region, and topic clusters. Set floors or caps per group where the target requires balance. Review a sample from each group, not only the top of the ranking.

1. Variants

7.1 Streaming corpus

When new data arrives daily, run the funnel incrementally: filter, dedup against everything seen so far, embed, and score only the new items. Rebuild the training-set index when the training set changes.

7.2 Text-only data for language models

The same funnel applies. Quality filters use perplexity under a small model and classifiers for spam and boilerplate. Near-duplicate removal matters even more, because repeated text degrades training.

7.3 Finding rare failure cases

Invert the goal: find inputs where the current model fails. Score candidates by the model's uncertainty or by disagreement between two models, and send the top ones to labelers.

7.4 At ten times the corpus

At 10 billion images, embedding takes about 1,400 GPU-hours and the quantized index about 640 GB. Shard the ANN index across machines, and compute novelty against a sample of the training set's neighborhoods instead of the whole set where recall allows. Deduplication moves to a distributed job keyed by LSH buckets.

1. The transferable pattern

This is a selection funnel over embeddings: define measurable scores, spend cheap computation on everything and expensive computation on few, put humans where they add the most information, and report precision and recall with honest margins. The same funnel drives content moderation queues, fraud review, and search relevance labeling.

Review: the 30-second answer

• Define novel and useful as scores. Distance to training data, times quality, times relevance.
• Funnel. Filters and dedup on everything, one embedding pass, expensive checks on survivors.
• Novelty rewards junk unless you stop it. Quality first, prefer dense new clusters, review each score band.
• Object search is retrieval, verification, and active learning. Text search, detector, labels, classifier, repeat.
• Measure. Sampled precision, estimated recall, and downstream model evaluation.

Quiz

Sources and further reading

• Learning Transferable Visual Models From Natural Language Supervision (CLIP) introduced the image-text embeddings used for text-to-image search.
• Deduplicating Training Data Makes Language Models Better measures the effect of near-duplicate removal.
• Billion-scale similarity search with GPUs (FAISS) covers product quantization and large ANN indexes.
• Mining of Massive Datasets, chapter 3 explains MinHash and locality-sensitive hashing.
• The measurement discipline here is the same as building a golden set in Evaluations. Embedding details are in embedding and search design.