Comparison
Adaptive query executionvsCardinality estimate
Adaptive query execution
the engine started the join, saw that one side was much smaller than predicted, and switched strategy mid-query.
Re-planning parts of a query at runtime using actual statistics from completed stages, rather than committing to the plan made up front. It exists because estimates on raw files are frequently terrible, and it fixes exactly the cases that hurt most: a mis-sized broadcast, a skewed partition, an over-parallelised stage. It also makes performance less predictable run to run, which is worth knowing when a query's timing suddenly changes with no other cause.
Full entry →Cardinality estimate
the plan says it expects nine hundred rows from that filter, it actually returns forty million, and every choice after that is wrong.
The planner's guess at how many rows each step will produce, from which it picks join algorithms, join order and memory allocations. A bad estimate early in the plan compounds through everything after it, which is why one wrong filter estimate can make an otherwise sensible query catastrophic. Comparing estimated against actual row counts in the plan is the fastest way to find where a query went wrong.
Full entry →