Notes on the architecture of data
Designing the structures that make data trustworthy.
Essays and field notes on data architecture, data modeling, dimensional modeling, data contracts, and the lakehouse — practical writing for data engineers and architects who design systems that make information trustworthy.
Most read start with these
Field Notes
Kimball vs Inmon: Two Ways to Build a Data Warehouse
Kimball and Inmon differ on one decision: build dimensional marts bottom-up, or a normalized enterprise warehouse top-down. The trade-off, and the modern blend.
★Field Notes
Star Schema vs Snowflake Schema: Which to Use and When
Star schema vs snowflake schema comes down to one decision: whether you normalize your dimensions. The difference, a worked example, and why the star wins.
★Field Notes
Data Lake vs Lakehouse: What Changed and Which to Use
A data lake stores raw files cheaply and guarantees nothing. A lakehouse adds a table layer over the same files for ACID, schema, and trust. When each fits.
★Field Notes
How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie
The table format war is settled; the catalog decides governance and lock-in now. How Iceberg catalogs work, compared honestly — and a decision rule that holds.
★Field Notes
One Big Table vs the Star Schema: The Real Trade-off
One Big Table denormalizes everything into one wide table; the star schema keeps facts and dimensions apart. What each costs, and why the answer is usually both.
★Field Notes
What Is a Data Catalog, and Do You Need One?
A data catalog is a searchable inventory of your data: what exists, where it lives, what it means, who owns it. What it's for, and when you actually need one.
Essays & field notes 47 pieces
Measured
Parquet vs ORC vs Avro: I Ran the Benchmark, and One Result Surprised Me
Real numbers on 3 million rows: compression, write time, and column-pruned reads across Parquet, ORC, Avro, and CSV. The script is published; rerun it.
02Field Notes
Merge-on-Read vs Copy-on-Write: The Lakehouse Decision With Numbers
Copy-on-write rewrites whole files on update; merge-on-read annotates them and resolves at read time. The trade is write cost against read cost, with arithmetic.
03Field Notes
Knowledge Graph vs Data Warehouse: Aggregation vs Connection
A warehouse answers how much; a knowledge graph answers how things connect. They're different query shapes, not rivals — and most serious estates end up with both.
04Field Notes
Ontology vs Data Model: Meaning You Can Reuse vs Structure You Can Query
A data model shapes data for one system; an ontology defines what concepts mean, independent of any system — and machines now read your schema.
05Field Notes
Parquet vs ORC vs Avro: Which Data File Format, and When
Parquet and ORC are columnar formats for analytics; Avro is row-based for streaming and schema evolution. A clear comparison and a decision rule that holds.
06Field Notes
Apache Iceberg vs Parquet: They're Not the Same Layer
Iceberg and Parquet aren't rivals — Parquet is a file format, Iceberg is a table format that manages Parquet files. Here's the difference, and why it matters.
07Field Notes
What Is Reverse ETL? The Warehouse Learns to Talk Back
Reverse ETL syncs modelled warehouse data back into CRMs, ad platforms, and support tools — so business teams act on it where they work. How it works, and its risks.
08Field Notes
How to Build a Data Catalog That People Actually Use
A tool-neutral guide to building a data catalog: start from query logs, document the fifty tables that matter, assign owners, automate the rest — in that order.
09Field Notes
The Inmon Methodology: The Corporate Information Factory, with an Example
Inmon's method builds one normalized enterprise warehouse first, then serves dimensional marts from it. How the CIF works, a worked example, and where it fits today.
10Field Notes
Data Observability vs Data Quality: What Each One Actually Catches
Data quality checks the rules you wrote; data observability watches for the failures you didn't anticipate. Neither replaces the other — here's the honest split.
11Field Notes
How to Choose an Iceberg Catalog: Unity vs Polaris vs Glue vs Nessie
The table format war is settled; the catalog decides governance and lock-in now. How Iceberg catalogs work, compared honestly — and a decision rule that holds.
12Field Notes
Iceberg vs Delta Lake: How to Actually Choose in 2026
Iceberg and Delta Lake have converged on capabilities — ACID, time travel, deletion vectors. The real decision is your platform and catalog, not the format.
13Field Notes
The Grain of a Fact Table: The First Decision That Decides Everything Else
The grain is the business definition of what one fact table row represents. Declare it first — every dimension, measure, and bug traces back to it.
14Field Notes
What Is an Open Table Format? Iceberg, Delta, and Hudi Explained
An open table format is a metadata spec that turns raw files in object storage into real tables — with ACID transactions, schema evolution, and time travel.
15Field Notes
Normalization vs Denormalization: When Each Wins
Normalization splits data into many tables to remove redundancy; denormalization combines them to remove joins. One favors writes, the other reads. The trade.
16Field Notes
Data Lake vs Lakehouse: What Changed and Which to Use
A data lake stores raw files cheaply and guarantees nothing. A lakehouse adds a table layer over the same files for ACID, schema, and trust. When each fits.
17Field Notes
Data Engineer vs Data Architect vs Analytics Engineer: Who Does What?
Three overlapping data roles, three different jobs: engineers move the data, analytics engineers shape it, architects design the system. The real division.
18Field Notes
ETL vs ELT: The Same Three Letters in a Different Order
ETL transforms data before loading it; ELT loads raw data first and transforms it in the warehouse. A trivial-sounding reorder that encodes the cloud shift.
19Field Notes
Kimball vs Inmon: Two Ways to Build a Data Warehouse
Kimball and Inmon differ on one decision: build dimensional marts bottom-up, or a normalized enterprise warehouse top-down. The trade-off, and the modern blend.
20Field Notes
One Big Table vs the Star Schema: The Real Trade-off
One Big Table denormalizes everything into one wide table; the star schema keeps facts and dimensions apart. What each costs, and why the answer is usually both.
21Field Notes
Batch vs Streaming: How to Actually Decide
Batch vs streaming isn't legacy vs modern. The real question is what latency the decision actually requires. Default to batch; promote one use case at a time.
22Field Notes
What Is Data Lineage, and What Is It Actually For?
Data lineage records where data comes from, how it's transformed, and where it goes. What it's genuinely for, impact analysis and root cause, and its limits.
23Field Notes
What Is Change Data Capture (CDC), and When Do You Need It?
Change data capture identifies inserts, updates, and deletes in a source database and delivers them downstream. How the three methods compare, and when batch wins.
24Reconsidered
Is the Modern Data Stack Dead?
The modern data stack isn't dead, but the era of bolting together a dozen best-of-breed SaaS tools is ending. What's actually happening, and what to keep.
25Field Notes
What Is a Vector Database, and Do You Need One?
A vector database stores embeddings and finds items by meaning rather than exact match. What that's for, how it fits your stack, and when a dedicated one pays.
26Field Notes
Data Vault vs Dimensional Modeling: Which Belongs Where
Data Vault and dimensional modeling aren't rivals; they solve different problems at different layers. What Data Vault costs, and when it's worth paying for.
27Essay
What Does a Data Architect Actually Do?
A data architect's job isn't drawing diagrams or picking tools. It's deciding how data is shaped, defined, owned, and trusted, then defending those decisions.
28Field Notes
What Is a Data Catalog, and Do You Need One?
A data catalog is a searchable inventory of your data: what exists, where it lives, what it means, who owns it. What it's for, and when you actually need one.
29Essay
Most Data Quality Problems Are Org-Chart Problems
When data is wrong, teams buy a tool. But the durable causes are organizational: unclear ownership, misaligned incentives, no accountability. Tools miss those.
30Field Notes
Fact Table Types: Transaction, Periodic Snapshot, and Accumulating
Three kinds of fact table, distinguished by what one row represents over time: transaction, periodic snapshot, and accumulating snapshot. How to pick one.
31Essay
Your AI Is Only as Good as Your Data Architecture
RAG, AI agents, and LLMs querying your warehouse are only as reliable as the data beneath them. GenAI raises the price of getting the architecture wrong.
32Reconsidered
What GenAI Actually Changes About Data Architecture — and What It Doesn't
GenAI adds vector storage and new retrieval patterns to the data stack, but structure, quality, governance, and ownership matter more than ever, not less.
33Field Notes
Slowly Changing Dimensions, Explained Without the Jargon
Slowly changing dimensions answer one question: when an attribute changes, do you overwrite history or preserve it? SCD Types 1, 2, and 3, and when to use each.
34Field Notes
Data Warehouse vs Data Lake vs Lakehouse: A Clear Comparison
A data warehouse stores structured, modeled data. A data lake stores raw data cheaply. A lakehouse tries to be both. A side-by-side, and how to choose.
35Field Notes
How to Make a Data Pipeline Idempotent
An idempotent pipeline gives the same result whether it runs once or five times. The concrete patterns: partition overwrite, merge on keys, delete-insert.
36Essay
What Is a Semantic Layer, and Why Does Your Data Stack Need One?
A semantic layer is the single governed place where business metrics are defined once, independent of any dashboard. What it is, and what it actually fixes.
37Reconsidered
The Medallion Architecture, Reconsidered
Bronze, silver, gold is a useful default and a dangerous dogma. A second look at what the layers get right, and where they quietly fall apart.
38Field Notes
Factless Fact Tables, Explained
A factless fact table records that an event happened, or could have, without any numeric measure. Why a fact table with no facts is useful, and the two types.
39Field Notes
OLTP vs OLAP: Why You Shouldn't Run Analytics on Your App Database
OLTP handles many small transactions fast; OLAP scans huge volumes for analysis. Opposite optimizations, which is why analytics on production is a trap.
40Field Notes
What Are Conformed Dimensions, and Why Do They Matter?
A conformed dimension is one dimension shared identically across several fact tables, so different business processes can be compared on the same terms.
41Essay
Data Contracts Are a Cultural Problem
A schema check is the easy 10% of a data contract. The other 90% is an organizational agreement that no YAML file can enforce for you.
42Field Notes
The Date Dimension: How to Build One and Why You Need It
A date dimension is a table with one row per calendar day, pre-loaded with every attribute of that day. Why every warehouse needs one, and how to build it.
43Field Notes
Star Schema vs Snowflake Schema: Which to Use and When
Star schema vs snowflake schema comes down to one decision: whether you normalize your dimensions. The difference, a worked example, and why the star wins.
44Field Notes
Fact Table vs Dimension Table: The Core Distinction
A fact table stores the measurements you analyze; a dimension table stores the context you analyze them by. The distinction every dimensional model rests on.
45Field Notes
A Field Guide to Dimensional Modeling
Facts, dimensions, and grain — the three ideas that quietly run most analytics, explained without the dogma.
46Field Notes
Surrogate Keys vs Natural Keys: A Practical Rule
Surrogate key vs natural key is a decision every data model faces. The practical rule: surrogate keys for dimensions, natural key kept as an attribute.
47Manifesto
The Shape of Data
Every dataset has a shape. The only question is whether you chose it, or whether it happened to you.
Essays by email
One new essay on data architecture, straight to your inbox. No noise, unsubscribe anytime.