jargon

Data engineering·Files, formats and table layers

there is a metadata layer over the files that says exactly which files are in the table right now, so two writers no longer overwrite each other.

Open table format

Also calledtable format, transactional table format

A specification that turns a directory of files into a table with atomic commits, snapshot isolation and schema evolution, by keeping an explicit manifest of the files each version contains. It is what removes the lake's oldest failure — readers seeing a half-written table — and it makes deletes and updates expressible without rewriting everything. Because the metadata is the table, its maintenance becomes something you have to run.

Commonly confused with