Comparison
Manifest filevsOpen table format
Manifest file
the engine reads one metadata file listing the table's data files and their statistics, instead of listing a bucket prefix.
The metadata object recording which data files belong to a table version, together with per-file statistics. Replacing directory listing with a read of this file is what makes table formats fast on object storage and what makes their commits atomic — swapping one pointer changes the whole table. It also means metadata itself accumulates and needs maintenance, which is a class of operational problem that plain directories did not have.
Full entry →Open table format
there is a metadata layer over the files that says exactly which files are in the table right now, so two writers no longer overwrite each other.
A specification that turns a directory of files into a table with atomic commits, snapshot isolation and schema evolution, by keeping an explicit manifest of the files each version contains. It is what removes the lake's oldest failure — readers seeing a half-written table — and it makes deletes and updates expressible without rewriting everything. Because the metadata is the table, its maintenance becomes something you have to run.
Full entry →