Anthropic Interview: Design a Data Platform with Sensitivity Tiers

Part of the Anthropic system design question bank. The question is representative of the round. The analysis and solution are this site's own.

Problem statement

The platform collects events from products and services, lands them safely, cleans and joins them, and serves them to analysts, ML teams, and a few online services. Every dataset carries a sensitivity level, and access follows it. Specialized ML knowledge is not needed.

Clarifying questions

• What are the sources? Product events from apps, service logs, and nightly exports from production databases.
• How fresh must data be? Minutes for operational dashboards, a day for most analysis.
• Who reads, and how? Analysts with SQL, ML teams reading large files in bulk, and a few services that need millisecond lookups of derived features.
• What sensitivity levels exist? For practice: public, internal, personal, and restricted, where restricted includes customer content.
• What regulations apply? Deletion requests must be honored everywhere within a set time, such as 30 days.
• What volume? For practice: 200,000 events per second at 1 KB each.

What makes a data platform hard

Collecting data is easy. Three things make it hard.

Data changes shape. Producers add fields, rename them, and send malformed records. The platform must accept change without breaking every downstream job.

Data is wrong sometimes. A buggy producer or a bad transform writes nonsense, and every table built on it inherits the damage. The platform must find the damage, contain it, and rebuild.

And data is sensitive. A personal field copied into a new table through a join keeps its sensitivity even when nobody remembers where it came from. Access control that only protects the raw tables leaks through derived ones.

So the driving tension is openness versus control. Teams need broad, fast access to do their jobs, while every sensitive field must stay protected through every copy and transform.

flowchart LR
  S[Sources]:::user --> I[Ingest]:::svc --> RZ[(Raw zone)]:::store
  RZ --> CZ[(Clean zone<br/>labeled)]:::store --> CUR[(Curated zone)]:::store
  CUR --> Q[SQL / files / serving]:::svc
  P[Policy layer<br/>labels + rules + audit]:::bad -.-> CZ
  P -.-> CUR
  P -.-> Q
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

Key idea. Treat sensitivity as data that flows with the data. Labels assigned at ingestion must survive every join and copy, or access control leaks.

Key concepts

Zones

Data moves through zones with rising quality and falling sensitivity. The raw zone holds exactly what arrived, append-only and tightly restricted. The clean zone holds validated, typed, labeled tables. The curated zone holds tables built for specific teams, often with sensitive fields masked.

Batch and streaming

Batch processing runs over a bounded chunk of data, such as one day. Streaming processes records continuously as they arrive. The classic Lambda architecture runs both paths side by side and merges them. The Kappa architecture runs everything as a stream and replays the log to recompute. Modern engines can run the same code in both modes, which removes the duplicate logic that made Lambda painful.

Lakehouse

A lakehouse stores tables as columnar files in object storage, with a table format that adds transactions, schema evolution, and time travel. It gives warehouse-like tables at object-storage cost, and many engines can read the same data.

Access control models

Role-based access control (RBAC) grants permissions to roles, such as "analytics team can read tables in the sales domain." Attribute-based access control (ABAC) decides from attributes of the user, the data, and the request, such as "users in region EU can read rows where region is EU, and personal columns only with a purpose grant." ABAC expresses sensitivity rules that RBAC cannot.

Lineage

Lineage records which inputs produced each output, down to the column. It answers two questions the platform cannot live without: what did this bad input touch, and where did this personal field end up?

Key idea. Zones separate trust levels, one engine handles batch and stream, a lakehouse stores it cheaply, ABAC expresses sensitivity, and lineage makes both containment and compliance possible.

1. Requirements

Before reading on. List the requirements, then name the property you would protect first and the constraint that shapes the design.

1.1 Functional requirements

• Ingest streaming events and batch exports.
• Validate records against schemas and route failures aside.
• Process into cleaned and curated tables on schedules and continuously.
• Serve SQL, bulk file access, and low-latency lookups.
• Label every column with a sensitivity level and enforce access by it.
• Honor deletion requests across every copy.
• Record lineage and an audit log of every read of sensitive data.

1.2 Non-functional requirements

• Freshness. Minutes for streaming tables, a day for batch.
• Correctness. Reruns give identical results; late data is handled by a stated rule.
• Cost. Storage and compute scale with use, with cold data in cheap storage.
• Isolation. A heavy backfill cannot starve daily jobs.

1.3 The constraint versus the property

Protection of sensitive data is the property. A leak of personal or customer data is the one failure the platform cannot undo. Data volume is the constraint. At 17 TB a day of raw input, every design choice about storage format, partitioning, and compute has a cost.

1. Back-of-the-envelope estimation

2.1 Ingest

200,000 events per second × 1 KB = 200 MB/s. Over a day: 200 MB/s × 86,400 s ≈ 17 TB raw.

2.2 Stored

Columnar formats compress event data well. Assume 5 times, so about 3.5 TB a day in the clean zone, or about 1.3 PB a year before replication. With object storage replicating internally, that is the number to budget.

2.3 Hot versus cold

If 90 days are hot and the rest cold, hot data is about 90 × 3.5 TB = 315 TB. The remaining year is about 960 TB in a colder storage class that costs a fraction as much per byte.

2.4 Query cost

A query that scans one day of one event table reads 3.5 TB at most, less if it filters on partition columns and reads only a few columns. Partition by date, then by a common filter such as product or region. A query that forgets the date filter scans a year: put a guard in the query engine that rejects unbounded scans on large tables.

Key idea. 17 TB a day raw, 3.5 TB stored, 1.3 PB a year. Partitioning and columnar reads decide what queries cost.

1. API design

The platform's interfaces are contracts more than endpoints.

3.1 Producer contract

POST /v1/events  (batched)
  [{event_id, event_type, schema_version, occurred_at, producer, payload}]
  -> 202 {accepted, rejected: [{event_id, reason}]}

schema registry
  PUT /v1/schemas/:event_type/versions   {schema, compatibility: backward}
  -> 409 if the new schema breaks backward compatibility

3.2 Access requests

POST /v1/access-requests  {dataset, columns?, purpose, duration}
  -> pending approval by the data owner
GET  /v1/datasets/:name   -> {owner, schema, column_labels, lineage_upstream, freshness}

3.3 Deletion

POST /v1/deletion-requests  {subject_id, reason}
  -> {request_id, due_by}
GET  /v1/deletion-requests/:id -> {status per dataset}

1. Data model

The platform's own metadata is as important as the data.

dataset        (name, zone, owner_team, format, partition_spec, retention)
column         (dataset, name, type, sensitivity: public|internal|personal|restricted,
                tags: [email, phone, customer_content, ...])
lineage_edge   (output_dataset, output_column, input_dataset, input_column, job_id)
policy         (id, subject: role|attribute expr, dataset pattern,
                action: read|write, row_filter?, column_rule: allow|mask|hash|deny)
grant          (user, dataset, purpose, approved_by, expires_at)
audit_log      (ts, user, dataset, columns_read, row_filter_applied, query_id)
job_run        (job_id, partition, input_versions, output_version, status, checks)

input_versions on each job run records which snapshot of each input a partition was built from. With table time travel, that makes any partition reproducible.

1. High-level design

5.1 Everyone reads the production database

flowchart LR
  A[Analysts]:::user --> PDB[(Production DB)]:::bad
  M[ML team]:::user --> PDB
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

Heavy queries slow the product, nobody can see history beyond what the product keeps, and every analyst can read every column. It fails on load, on history, and on access.

5.2 Fix 1: a raw zone fed by ingestion

Events go through an ingestion service into a durable log, and a loader writes them to append-only files in object storage. Database exports land there too. Production is no longer queried directly.

flowchart LR
  P[Products]:::user --> IN[Ingest API]:::new --> LOG[[Durable log]]:::new --> RAW[(Raw zone)]:::new
  DB[(Prod DB exports)]:::store --> RAW
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

5.3 Fix 2: validate, type, and label into a clean zone

A processing job reads the raw zone, validates each record against its registered schema, routes failures to a dead-letter area with the reason, and writes typed tables. A classifier and the schema's own tags assign a sensitivity label to every column.

5.4 Fix 3: curated tables and a policy layer

Team-facing tables are built from clean tables. Every query engine asks one policy service before it runs a query: which rows, which columns, masked or clear. Every read of personal or restricted data is logged.

flowchart LR
  U([User]):::user --> QE[Query engine]:::svc
  QE -->|who, dataset, columns, purpose| PS[Policy service]:::new
  PS -->|row filter + column rules| QE
  QE --> T[(Tables)]:::store
  QE -->|audit record| AL[(Audit log)]:::new
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef new fill:#ffffff,stroke:#c4492d,stroke-width:2px,stroke-dasharray:5 3,color:#171717;

5.5 Fix 4: lineage and label propagation

Every job reports its column-level inputs and outputs. The catalog uses them for lineage, and it computes each output column's label as the highest label among its inputs, unless the transform declares that it masks or aggregates the field. A column built by joining an email address stays personal until a job explicitly hashes it.

5.6 The composed design

flowchart TB
  SRC[Sources]:::user --> IN[Ingest + schema registry]:::svc --> LOG[[Log]]:::svc --> RAW[(Raw: append only, restricted)]:::store
  RAW --> VAL[Validate, type, classify]:::svc --> CLEAN[(Clean: labeled tables)]:::store
  VAL --> DLQ[(Dead letters)]:::bad
  CLEAN --> TR[Idempotent transforms]:::svc --> CUR[(Curated: team tables)]:::store
  CUR --> SQL[SQL engine]:::svc
  CUR --> FILES[Bulk file access]:::svc
  CUR --> FS[(Feature store / cache)]:::store
  CAT[Catalog: labels, lineage, policies]:::svc -.-> VAL
  CAT -.-> TR
  CAT -.-> SQL
  CAT -.-> FILES
  classDef user fill:#e6efec,stroke:#315e55,color:#171717;
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

Key idea. Land raw data safely, validate and label it once, transform idempotently, and route every read through one policy layer backed by a catalog with lineage.

1. Deep dives

6.1 Schema evolution without breakage

Before reading on. A producer renames user_id to account_id and deploys on a Friday. What stops every downstream job from failing?

The schema registry. Each event type has versioned schemas with a compatibility rule, typically backward compatible: new readers can read old data, and new fields must be optional. A rename is a breaking change, so the registry rejects it at the producer's build step. The producer instead adds account_id, keeps user_id for a deprecation period, and consumers migrate.

Records that fail validation anyway, from a producer that skipped the registry, go to the dead-letter area with the reason and the schema version. An alert fires to the owning team. The pipeline keeps running for everyone else.

6.2 Idempotent processing and late data

Before reading on. Yesterday's daily job ran twice because of a retry. Are the numbers now doubled?

They must not be. Make every transform write its output by partition, with overwrite semantics: rerunning the job for day D replaces day D's partition entirely. Since the inputs for day D did not change, the result is identical. Appending instead of overwriting is how numbers double.

Late data needs a rule. Events carry occurred_at, and a mobile client may send yesterday's events today. Streaming jobs use a watermark: they wait a set time, such as one hour, before treating a window as complete. Batch jobs reprocess the last few days' partitions each night to absorb stragglers. State the correction window and what happens to events later than that: they land in the current partition with a flag, or they are dropped and counted.

Deduplicate on event_id in the clean zone, because producers retry and the log delivers at least once.

6.3 Access control that survives joins

Before reading on. An analyst without personal-data access joins an orders table with a users table. The users table has an email column. What does the analyst see?

The analyst sees the email column masked, or the query is denied for that column, depending on policy. The policy service evaluates each column the query reads, not just each table.

The harder problem is derived tables. If a job writes a curated table that includes the email, that table must be labeled personal. Label propagation through lineage handles it: the output column inherits the highest input label. A job that hashes the email declares the transform, and its output column becomes internal.

Row-level rules go into the same policy. A regional team gets a row filter region = 'EU' applied automatically by the engine. Purpose grants handle exceptions: an investigator requests access to restricted data for a stated purpose and duration, an owner approves, and every read is logged against that grant.

flowchart LR
  E[users.email<br/>personal]:::bad --> J[join job]:::svc --> O[curated.orders_enriched.email<br/>personal, inherited]:::bad
  E --> H[hash job<br/>declares masking]:::svc --> O2[curated.orders.user_hash<br/>internal]:::store
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

6.4 Bad data and blast radius

Before reading on. A producer sent prices in cents instead of dollars for six hours. Revenue dashboards tripled. How do you find what is affected and fix it?

Prevent first. Run data checks after each stage: row counts within expected bounds, null rates, value ranges, and distribution shifts against the last week. A failing check blocks downstream jobs for that partition, so the damage stops at one table.

When damage gets through, lineage answers the scope question: every table and column downstream of the bad input for those hours. Fix the input, either by correcting the raw records or by adding a correction in the clean layer. Then rerun the affected partitions in dependency order. Idempotent overwrites make the rerun safe. Tell the owners of affected tables through the catalog, with the time range.

6.5 Backfills and isolation

Backfills rebuild months of partitions and can take a whole cluster. Run them in a separate compute pool with its own quota, at lower priority. Daily jobs keep their own pool. A backfill that competes with the daily run delays every dashboard in the company.

6.6 Deletion requests

A deletion request names a subject. Lineage lists every dataset holding that subject's data. For each, a deletion job rewrites the affected partitions without the subject's rows; the lakehouse table format supports row-level deletes as a transaction. Raw-zone retention for personal data is short, such as 30 days, so the raw copy ages out within the deadline. Snapshots used for time travel must also expire within the deadline, or they keep deleted data alive.

6.7 Serving three kinds of readers

Before reading on. Analysts run SQL, ML teams read terabytes of files, and a pricing service needs a user's features in 5 ms. Can one storage layer serve all three?

One copy of the data can, three access paths are needed.

Analysts use a distributed SQL engine over the lakehouse tables. Queries prune partitions and read only the columns they touch. Popular dashboards run on small, pre-aggregated tables refreshed hourly, so they do not rescan raw events. Put a query cost guard in front: reject queries on large tables without a date filter, and cap scanned bytes per query by role.

ML teams read files directly: the same columnar files, through the table format's reader, at full object-storage throughput. They get snapshots by version, so a training run can pin exactly what it read. The policy layer still applies: exports go through a job that applies column rules before writing the files the team can read.

Online services cannot query a lakehouse in 5 ms. A streaming job computes the features they need and writes them to a key-value feature store, keyed by user or entity. The same feature definition runs in batch to produce training data, so values match between training and serving.

flowchart LR
  LH[(Lakehouse tables)]:::store --> SQL[SQL engine<br/>analysts]:::svc
  LH --> FILES[Snapshot reads<br/>ML training]:::svc
  STREAM[Streaming feature job]:::svc --> FS[(Feature store<br/>ms lookups)]:::store
  LOG[[Event log]]:::svc --> STREAM
  LOG --> LH
  POL[Policy layer]:::bad -.-> SQL
  POL -.-> FILES
  POL -.-> FS
  classDef svc fill:#f4f1e8,stroke:#315e55,color:#171717;
  classDef store fill:#fdf3dc,stroke:#c4492d,color:#171717;
  classDef bad fill:#fbe9e4,stroke:#c4492d,color:#171717;

6.8 Cost control

At 1.3 PB a year, storage and compute costs need active management.

• Storage tiers. Hot partitions on standard object storage, older ones on infrequent-access classes, and raw data expired on a schedule. Compaction merges small files into large ones, which cuts both storage overhead and query time.
• Compute quotas. Each team has a compute budget for queries and jobs, with reports that show cost per query and per scheduled job. The five most expensive jobs usually account for a large share of the bill and are easy to fix.
• Materialize wisely. A dashboard that scans a week of raw events every five minutes costs far more than an hourly aggregate table that it reads instead.
• Retire unused tables. The catalog records last read time per table. Tables unread for 90 days go to their owners for deletion, which also shrinks the deletion-request surface.

For practice: if a popular dashboard scans 25 TB (one week of clean data) 12 times an hour, that is 300 TB of scanning per hour. An hourly aggregate of a few gigabytes serves the same dashboard. Show this kind of arithmetic; it is how data platforms actually save money.

1. Variants

7.1 Real-time features for online services

A service needs a user's purchase count in the last hour, in milliseconds. Compute it in a streaming job and write it to a key-value feature store keyed by user. The same definition runs in batch for training data, so online and offline values match.

7.2 Multi-region data residency

EU data stays in the EU. Run ingestion and storage per region. Cross-region queries see only aggregated or anonymized outputs that policy allows to leave the region.

7.3 Training data for ML

ML teams want large snapshots of curated data. Give them versioned snapshots with the policy applied at export time, and record the snapshot ID with every training run, so a later deletion request can find which models used the subject's data.

7.4 At ten times the volume

At 2 million events per second, ingestion moves to many log partitions and many loader workers, but the architecture holds. Three things get harder. The catalog and lineage graph grow to millions of columns, so lineage queries need their own indexed store. Deletion requests touch far more partitions, so batch them daily per dataset. And the daily batch window tightens, so more pipelines move to incremental processing that only handles new partitions.

1. The transferable pattern

A data platform is a pipeline of trust zones with metadata that travels. The data moves from untrusted to curated, and its labels, schemas, and lineage move with it. The same pattern governs any system where copies multiply: backups, caches of user data, search indexes, and training sets. Wherever data is copied, ask what metadata must be copied with it.

Review: the 30-second answer

• Raw, clean, curated zones. Append-only raw data, validated and labeled clean tables, team-facing curated tables.
• Schema registry. Breaking changes rejected at the producer; bad records dead-lettered.
• Idempotent partition overwrites. Reruns and backfills are safe.
• Labels follow lineage. One policy layer applies row filters and column masks, and audits sensitive reads.
• Checks and blast radius. Stage checks stop bad data; lineage scopes and reruns repair it.

Quiz

Sources and further reading

• Apache Iceberg documents an open table format with schema evolution, row-level deletes, and time travel.
• Questioning the Lambda Architecture by Jay Kreps introduces the Kappa approach of reprocessing from the log.
• OpenLineage is an open standard for collecting column-level lineage from jobs.
• Security and compliance covers classification, access, and retention for customer data.