Data engineering·Files, formats and table layers
there are no directories really, listing a prefix with a million keys is slow, and you cannot append to a file.
Object storage
Also calledblob storage, S3-compatible storage
Storage addressed by key rather than by path, with immutable objects and eventual listing behaviour. Almost every awkward thing about lake engineering comes from this: renames are copies, directory listings are expensive, partial updates are impossible, so a 'table' is a set of whole files and changing a row means writing a new file. Understanding that constraint explains compaction, manifests and copy-on-write in one go.