Data engineering·Cost, layout and query performance
the join needs matching keys on the same machine, so the engine moves the whole dataset across the network before it can start.
Shuffle
Also calledexchange, repartition, data movement
Redistributing data across workers so that rows sharing a key end up together, which joins, group-bys and window functions all require. It is the expensive step in any distributed query, because it is bounded by network and disk rather than by CPU. Most distributed query tuning is really about avoiding shuffles, reducing what they carry, or making sure the data was already partitioned the way the join needs.