Data engineering·Cost, layout and query performance
both sides are too big to broadcast, so each is shuffled and sorted by the key and then walked through in step.
Sort-merge join
Also calledshuffle sort merge join
Joining two large datasets by partitioning both on the key, sorting each partition, and merging. It scales to inputs of any size and it is the expensive default, because it pays for both a shuffle and a sort. It is also the join most affected by skew, since a single heavy key produces one partition that dominates the runtime.