Files, formats and table layers

everything anyone might ever need is in a bucket as files, in whatever shape it arrived, and finding the useful part is your problem.

Data lake

the data is in typed, governed tables you query in SQL, and getting anything in means agreeing a schema first.

Data warehouse

the tables are open files in a bucket and you still get transactions, schema enforcement and a rollback.

Lakehouse

there are nine thousand tables in the bucket, four of them are used, and nobody knows which four.

Data swamp

there are no directories really, listing a prefix with a million keys is slow, and you cannot append to a file.

Object storage

the query reads three columns out of two hundred and only touches the bytes for those three.

Columnar format

the whole record is written together, so appending one event is cheap and reading one field of a billion records is not.

Row-oriented format

the files are columnar with statistics in the footer, so the engine can skip whole chunks without reading them.

Parquet

the tables are in the other columnar format, the one that came out of the Hive world, with stripes instead of row groups.

ORC

each message carries a schema id rather than field names, and adding an optional field does not break the consumers already running.

Avro

the file is split into blocks of a hundred and twenty-eight megabytes, each with its own min and max per column.

Row group

the column holds four distinct country names in eleven million rows, and the file stores small integers plus a lookup.

Dictionary encoding

someone switched the codec and the files got a third smaller while the queries got slightly slower to decode.

Compression codec

the where-clause is handed to the file reader, which uses the min and max in each chunk's footer to skip most of them unread.

Predicate pushdown

you replaced select star with the six columns you needed and the bytes scanned fell by ninety percent.

Projection pushdown

you filtered on the partition column and the engine opened four folders instead of nine hundred.

Partition pruning

the partition value is encoded in the folder name as dt=2026-01-14, and the column does not exist inside the files at all.

Hive-style partitioning

a streaming job wrote a file every ten seconds and now a single day's query opens eight thousand objects to read four hundred megabytes.

Small files problem

a maintenance job rewrites thousands of small files into a few large sorted ones, and nothing about the table's contents changes.

Compaction

there is a metadata layer over the files that says exactly which files are in the table right now, so two writers no longer overwrite each other.

Open table format

the engine reads one metadata file listing the table's data files and their statistics, instead of listing a bucket prefix.

Manifest file

the table's partitioning was changed without rewriting any of the existing files, and old queries kept working.

Iceberg

there is a _delta_log folder of JSON commits next to the Parquet, and reading the table means replaying that log.

Delta Lake

the table is built for frequent upserts by key, keeping a log of changes beside the base files and merging them on read.

Hudi

updating one row rewrites the entire file that row lived in, so writes are slow and every read is as fast as a plain scan.

Copy-on-write table

the update is written as a small delta file and the reader merges it with the base file every time, until a compaction folds them together.

Merge-on-read table

you queried the table as it stood before this morning's bad run, without restoring anything.

Time travel

you deleted the rows a month ago and the bytes were still in the bucket, because the old snapshots still referenced them.

Snapshot expiry

the file was written as whatever the source sent and the shape is decided by the query that reads it, which is why two teams parse it differently.

Schema-on-read

the load rejected the file because a column changed type, and the bad data never got into the table.

Schema-on-write

the vendor added three columns and renamed one over the weekend, and nobody told anybody.

Schema drift

the warehouse queries files it does not own, so dropping the table leaves the data exactly where it was.

External table