Data engineering·topic 3 of 8
Modelling for analysis
A warehouse schema is a user interface for people writing SQL under time pressure. The vocabulary here is thirty years old and still the fastest way to settle an argument about why a dashboard is double-counting.
Read in order · tick what you already know
- 01
you stopped copying the application's tables into the warehouse and started shaping it around what happened and what it happened to.
Dimensional modelling
- 02
the table has one row per order line, a handful of numbers you can add up, and the rest is foreign keys.
Fact table
- 03
everything you would ever put in a group-by or a filter — country, product name, channel — lives in one wide, short table.
Dimension table
- 04
someone asked what one row means and three people in the room gave three different answers.
Grain
- 05
you joined two fact tables at different grains and every revenue figure quietly doubled.
Fan trap
- 06
every dimension is one join away from the fact, so no query needs more than a single hop to get a label.
Star schema
- 07
getting the product's category name means joining the fact to product, then product to subcategory, then subcategory to category.
Snowflake schema
- 08
marketing and finance both group by the same customer dimension, so their two reports can finally be put side by side.
Conformed dimension
- 09
the dimension row has a meaningless integer id of your own invention, which is why the source renumbering its customers did not break anything.
Surrogate key
- 10
you joined on the email address, and then two people changed their email and one of them was reused by a new account.
Natural key
- 11
a customer moved from Leeds to Bristol, and now you have to decide whether their orders from last year happened in Leeds or in Bristol.
Slowly changing dimension
- 12
you overwrite the old value, and last year's report now shows the customer as having always been in Bristol.
SCD type 1
- 13
the change closes the old dimension row with an end date and opens a new one, so the customer now has three rows and only one is current.
SCD type 2
- 14
the row has both current_region and previous_region, and the region before that is gone.
SCD type 3
- 15
the order arrived before the customer record did, so the fact points at a placeholder row labelled unknown.
Late-arriving dimension
- 16
the order number sits on the fact table with no dimension table behind it, because there is nothing else to say about an order number.
Degenerate dimension
- 17
eight unrelated yes/no flags were collapsed into one small dimension of every combination, instead of eight columns on the fact.
Junk dimension
- 18
the same date dimension is joined three times as order date, ship date and return date, and the report is unreadable until you alias them.
Role-playing dimension
- 19
a bank account has three owners, so you cannot put an owner key on the fact without triple-counting the balance.
Bridge table
- 20
one row per thing that happened, inserted and never touched again, and the table only ever grows.
Transaction fact table
- 21
you write one row per account per day whether anything happened or not, so 'what was the balance on the fourteenth' is a lookup rather than a replay.
Periodic snapshot
- 22
the row for an order is updated as it moves through picked, packed and shipped, so each stage's date is a column.
Accumulating snapshot
- 23
the table records that a student attended a class, with no numbers on it at all, and the only measure is counting rows.
Factless fact table
- 24
you summed the daily account balances across the month and got a number thirty times too big.
Semi-additive measure
- 25
every attribute anyone might filter on is pre-joined onto the fact, so the analyst's query has no joins in it at all.
One big table
- 26
the table has four hundred columns, most queries read six of them, and nobody can tell you what the other three hundred and ninety-four are for.
Wide table
- 27
the finance team has its own set of tables, shaped for their questions, built from the shared ones.
Data mart
- 28
the same data exists three times: exactly as it arrived, cleaned and typed, and shaped for a specific report.
Medallion architecture
- 29
the raw layer is split into keys, relationships and attributes as three kinds of table, and nothing is ever updated or deleted.
Data vault