Data engineering·Cost, layout and query performance
you partitioned by customer id, there are two million customers, and now every query lists two million tiny folders.
Partitioning strategy
Also calledpartition design, partition column choice
The choice of what a table is physically split by, which sets how much data a filtered query can avoid reading. It is the highest-leverage physical decision in a warehouse and the hardest to change later, because changing it rewrites the table. The two failure modes are opposite and equally common: too coarse and nothing is pruned, too fine and you have the small files problem plus metadata that dwarfs the data.