Data engineering·Cost, layout and query performance
ninety-nine tasks finished in twenty seconds and the hundredth ran for eleven minutes, because one key has forty percent of the rows.
Data skew
Also calledpartition skew, key skew, hot partition
Work distributed unevenly across parallel tasks because the partitioning key is unevenly distributed. It is the single most common reason a distributed query is slow, and it is invisible in averages — the job's runtime is the slowest task's runtime. Nulls and default values are the classic culprits, because every row that failed to get an id ends up in the same partition.