Data engineering·Files, formats and table layers
there is a metadata layer over the files that says exactly which files are in the table right now, so two writers no longer overwrite each other.
Open table format
Also calledtable format, transactional table format
A specification that turns a directory of files into a table with atomic commits, snapshot isolation and schema evolution, by keeping an explicit manifest of the files each version contains. It is what removes the lake's oldest failure — readers seeing a half-written table — and it makes deletes and updates expressible without rewriting everything. Because the metadata is the table, its maintenance becomes something you have to run.