Comparison
Data skewvsSpill to disk
Data skew
ninety-nine tasks finished in twenty seconds and the hundredth ran for eleven minutes, because one key has forty percent of the rows.
Work distributed unevenly across parallel tasks because the partitioning key is unevenly distributed. It is the single most common reason a distributed query is slow, and it is invisible in averages — the job's runtime is the slowest task's runtime. Nulls and default values are the classic culprits, because every row that failed to get an id ends up in the same partition.
Full entry →Spill to disk
the sort did not fit in memory, started writing to local disk, and the query went from forty seconds to nineteen minutes.
An operator writing intermediate data to disk or to remote storage because it exceeded available memory. It is not an error, which is why it hides: the query still succeeds, just enormously slower. It is one of the first things to look for in a query that got dramatically worse without the SQL changing, and the usual causes are a data volume increase, a skewed key or a join order the planner got wrong.
Full entry →