jargon

Data engineering·Files, formats and table layers

there are no directories really, listing a prefix with a million keys is slow, and you cannot append to a file.

Object storage

Also calledblob storage, S3-compatible storage

Storage addressed by key rather than by path, with immutable objects and eventual listing behaviour. Almost every awkward thing about lake engineering comes from this: renames are copies, directory listings are expensive, partial updates are impossible, so a 'table' is a set of whole files and changing a row means writing a new file. Understanding that constraint explains compaction, manifests and copy-on-write in one go.

Commonly confused with