Data Engineering

the dashboard renders perfectly and every number on it is from Thursday.

Data freshness

someone asked what one row means and three people in the room gave three different answers.

Grain

two dashboards show different revenue because one includes refunds and neither of them says so anywhere.

Metric definition

the vendor added three columns and renamed one over the weekend, and nobody told anybody.

Schema drift

the query returned four rows and the console says it read two point one terabytes.

Bytes scanned

every task is green, the run took its usual eleven minutes, and it processed zero rows.

Silent failure

the report was wrong and the reason turned out to be six hops upstream in something nobody on your team owns.

Data pipeline

two systems disagree about the customer's address and you need to be able to say, without arguing, which one is allowed to be right.

System of record

the data was cleaned and reshaped by a job in the middle, and the warehouse only ever saw the tidy version.

ETL

you dumped the source table into the warehouse untouched and did all the shaping afterwards, in SQL, where everyone could read it.

ELT

nothing at all happened for twenty-three hours and then a job read the whole day at once.

Batch processing

records are handled one at a time as they land, and the job has been running without finishing for four months.

Stream processing

the job reads all four hundred million rows every night because nobody could work out which ones had changed.

Full load

each run asks the source only for rows changed since the last run, and one row updated in the same second as the cutoff is now missing forever.

Incremental load

the job stores the largest updated_at it has seen and asks for everything greater than that next time.

High-water mark

you tail the database's own replication log, so a delete and a mid-second update both come through without the source running a single extra query.

Log-based CDC

you re-ran the load after a failure and the rows updated in place instead of appearing twice.

Upsert

the same event is in the table three times because the connector retried, and you keep the latest by version per key.

Deduplication

you ran the same job three times after a failure and the output table is identical to running it once.

Idempotent pipeline

something has to decide that the transform runs only after all four extracts finished, and retry the one that did not.

Orchestrator

the pipeline is drawn as boxes with arrows, nothing points backwards, and a step waits for everything pointing at it.

DAG

the job is set to hourly, takes seventy minutes, and now two copies are running at once.

Schedule interval

the run that started at 02:00 on Tuesday is processing Monday's data, and half the confusion in the team is about which date the run is named after.

Logical date

the logic was wrong for three months, so you re-run the same job for ninety past days and hope nothing downstream notices mid-way.

Pipeline backfill

the file arrives in a bucket exactly as the vendor sent it, wrong headers and all, and nothing touches it there.

Landing zone

the supplier renamed a column overnight and you fixed it in one place instead of in forty downstream queries.

Staging layer

the same data exists three times: exactly as it arrived, cleaned and typed, and shaped for a specific report.

Medallion architecture

you stopped copying the application's tables into the warehouse and started shaping it around what happened and what it happened to.

Dimensional modelling

the table has one row per order line, a handful of numbers you can add up, and the rest is foreign keys.

Fact table

everything you would ever put in a group-by or a filter — country, product name, channel — lives in one wide, short table.

Dimension table

every dimension is one join away from the fact, so no query needs more than a single hop to get a label.

Star schema

the dimension row has a meaningless integer id of your own invention, which is why the source renumbering its customers did not break anything.

Surrogate key

you joined on the email address, and then two people changed their email and one of them was reused by a new account.

Natural key

a customer moved from Leeds to Bristol, and now you have to decide whether their orders from last year happened in Leeds or in Bristol.

Slowly changing dimension

the change closes the old dimension row with an end date and opens a new one, so the customer now has three rows and only one is current.

SCD type 2

there are no directories really, listing a prefix with a million keys is slow, and you cannot append to a file.

Object storage

everything anyone might ever need is in a bucket as files, in whatever shape it arrived, and finding the useful part is your problem.

Data lake

the data is in typed, governed tables you query in SQL, and getting anything in means agreeing a schema first.

Data warehouse

the tables are open files in a bucket and you still get transactions, schema enforcement and a rollback.

Lakehouse

the files are columnar with statistics in the footer, so the engine can skip whole chunks without reading them.

Parquet

the query reads three columns out of two hundred and only touches the bytes for those three.

Columnar format

you filtered on the partition column and the engine opened four folders instead of nine hundred.

Partition pruning

the phone was in a tunnel, so the tap happened at 14:02 and your job saw it at 16:40, and the hourly chart has to put it at 14:00.

Event time

yesterday's total went up this morning, because eight hundred events from a phone that was offline finally arrived.

Late-arriving data

the job announces that it believes nothing older than 14:30 is still coming, and closes every window that ended before then.

Watermark

nothing errored, every job is green, and the number on the board is wrong by eleven percent.

Data quality

the transform ships with a check that this column is never null, and the check runs against real output every night rather than against a fixture.

Data test

the column has been two percent null for a year, this morning it is sixty percent, and nothing else changed.

Null-rate check

the number is wrong and you need to know every table between it and the source, without reading forty SQL files.

Data lineage

the source team changed a status code's meaning, the pipeline kept running, and your funnel report has been wrong since Tuesday.

Upstream drift

the source team's deploy pipeline fails if they remove a field your models depend on, because the expectation is written down and checked.

Data contract

a free-text notes column turned out to contain names and phone numbers, and it had been copied into six downstream tables.

Personally identifiable information

one dashboard refreshing every five minutes turned out to be a third of the monthly warehouse bill.

Query cost

you partitioned by customer id, there are two million customers, and now every query lists two million tiny folders.

Partitioning strategy

ninety-nine tasks finished in twenty seconds and the hundredth ran for eleven minutes, because one key has forty percent of the rows.

Data skew