Moving data out of the source

the report was wrong and the reason turned out to be six hops upstream in something nobody on your team owns.

Data pipeline

the data was cleaned and reshaped by a job in the middle, and the warehouse only ever saw the tidy version.

ETL

you dumped the source table into the warehouse untouched and did all the shaping afterwards, in SQL, where everyone could read it.

ELT

the churn score computed in the warehouse gets pushed back into the CRM so a salesperson can see it in the tool they already use.

Reverse ETL

nothing at all happened for twenty-three hours and then a job read the whole day at once.

Batch processing

the 'streaming' job actually wakes every thirty seconds, reads whatever arrived, and writes one file.

Micro-batch

records are handled one at a time as they land, and the job has been running without finishing for four months.

Stream processing

the job reads all four hundred million rows every night because nobody could work out which ones had changed.

Full load

each run asks the source only for rows changed since the last run, and one row updated in the same second as the cutoff is now missing forever.

Incremental load

the job stores the largest updated_at it has seen and asks for everything greater than that next time.

High-water mark

the row is gone from the source and still in the warehouse, because your incremental load can only see rows that exist.

Hard delete detection

the delete arrives as a message with a key and no body, and a consumer that skips empty payloads keeps the row forever.

Tombstone record

you tail the database's own replication log, so a delete and a mid-second update both come through without the source running a single extra query.

Log-based CDC

you poll the source every few minutes with a where-clause on updated_at, and a row deleted outright simply stops appearing.

Query-based CDC

before the change feed is any use you have to copy the table as it stands today, while it is still being written to.

Initial snapshot

you re-ran the load after a failure and the rows updated in place instead of appearing twice.

Upsert

you stopped inserting a row at a time and pointed the warehouse at a folder of files, and the load went from hours to a minute.

Bulk load

the vendor posts to your endpoint whenever something changes, and when your endpoint was down for an hour those changes are simply gone.

Push-based ingestion

you ask the source for new rows on a schedule, so an outage on your side just means the next run reads more.

Pull-based ingestion

the sync broke because the vendor added a field, and the fix was in somebody else's code that you configure rather than edit.

Connector

six teams each built their own connector to the same production database and the DBA noticed before anyone else did.

Fan-out ingestion

the event carries the whole customer record rather than just an id, so the consumer never has to call back and ask.

Event-carried state transfer

the file arrives in a bucket exactly as the vendor sent it, wrong headers and all, and nothing touches it there.

Landing zone

the supplier renamed a column overnight and you fixed it in one place instead of in forty downstream queries.

Staging layer

two systems disagree about the customer's address and you need to be able to say, without arguing, which one is allowed to be right.

System of record

the row was committed in the source at 09:00 and appeared in the warehouse at 09:41, and nobody could say which hop ate the forty minutes.

Ingestion lag