Comparison
CompactionvsSmall files problem
Compaction
a maintenance job rewrites thousands of small files into a few large sorted ones, and nothing about the table's contents changes.
Rewriting a table's files into fewer, larger, better-ordered ones without changing the data. It is routine maintenance on any table written incrementally, and on a lakehouse it is your job rather than the storage engine's. It is not free — it rewrites data, it costs compute, and it interacts with time travel because the old files must be retained until their snapshots expire — so it is scheduled and budgeted like any other pipeline.
Full entry →Small files problem
a streaming job wrote a file every ten seconds and now a single day's query opens eight thousand objects to read four hundred megabytes.
Query cost dominated by per-file overhead — listing, opening, reading a footer — rather than by the data itself. It is the standard consequence of frequent writes to object storage, and it degrades gradually enough that it is usually noticed as 'the warehouse got slower' months later. The fix is compaction, and the prevention is making write frequency a deliberate decision rather than a side effect of the streaming interval.
Full entry →