Comparison
Data skewvsStraggler task
Data skew
ninety-nine tasks finished in twenty seconds and the hundredth ran for eleven minutes, because one key has forty percent of the rows.
Work distributed unevenly across parallel tasks because the partitioning key is unevenly distributed. It is the single most common reason a distributed query is slow, and it is invisible in averages — the job's runtime is the slowest task's runtime. Nulls and default values are the classic culprits, because every row that failed to get an id ends up in the same partition.
Full entry →Straggler task
the stage is at nine hundred and ninety-nine of a thousand tasks complete and has been for six minutes.
A single task holding up a stage while everything else has finished. The cause is either skew, where the task genuinely has more data, or an unhealthy node, where it does not. Distinguishing the two is the first diagnostic step, because the remedies are unrelated: one is a repartition or a salted key, the other is speculative execution or a dead machine.
Full entry →