Data engineering·Files, formats and table layers
a streaming job wrote a file every ten seconds and now a single day's query opens eight thousand objects to read four hundred megabytes.
Small files problem
Also calledtoo many small files, file fragmentation
Query cost dominated by per-file overhead — listing, opening, reading a footer — rather than by the data itself. It is the standard consequence of frequent writes to object storage, and it degrades gradually enough that it is usually noticed as 'the warehouse got slower' months later. The fix is compaction, and the prevention is making write frequency a deliberate decision rather than a side effect of the streaming interval.