Comparison
Partitioning strategyvsSharding
Partitioning strategy
you partitioned by customer id, there are two million customers, and now every query lists two million tiny folders.
The choice of what a table is physically split by, which sets how much data a filtered query can avoid reading. It is the highest-leverage physical decision in a warehouse and the hardest to change later, because changing it rewrites the table. The two failure modes are opposite and equally common: too coarse and nothing is pruned, too fine and you have the small files problem plus metadata that dwarfs the data.
Full entry →Sharding
one database stopped being enough, so you split the rows across several of them and now every query has to know which one to ask.
Splitting data across independent database instances so writes and storage scale past one machine. It is the point at which cross-shard joins, distributed transactions and global uniqueness stop being free. Resharding a live system is one of the hardest routine operations in backend engineering, which is why the shard key deserves more thought than the schema.
Full entry →