Comparison
Object storagevsSmall files problem
Object storage
there are no directories really, listing a prefix with a million keys is slow, and you cannot append to a file.
Storage addressed by key rather than by path, with immutable objects and eventual listing behaviour. Almost every awkward thing about lake engineering comes from this: renames are copies, directory listings are expensive, partial updates are impossible, so a 'table' is a set of whole files and changing a row means writing a new file. Understanding that constraint explains compaction, manifests and copy-on-write in one go.
Full entry →Small files problem
a streaming job wrote a file every ten seconds and now a single day's query opens eight thousand objects to read four hundred megabytes.
Query cost dominated by per-file overhead — listing, opening, reading a footer — rather than by the data itself. It is the standard consequence of frequent writes to object storage, and it degrades gradually enough that it is usually noticed as 'the warehouse got slower' months later. The fix is compaction, and the prevention is making write frequency a deliberate decision rather than a side effect of the streaming interval.
Full entry →