Data skew

What is data skew?

Data skew is when the rows in a dataset spread unevenly across the partitions that a distributed engine processes in parallel. One partition holds most of the rows, so a single task does most of the work while the rest of the cluster sits idle waiting for it.

Picture twenty supermarket checkout lanes where everyone queues at the same one: nineteen cashiers stand idle, a new lane changes nothing, and the shop runs at the speed of that single queue, the skewed partition. The values are fine; skew is about how they land when an engine like Apache Spark splits work by a column and expects each part to be about the same size. Partitioning is that deliberate split, and skew is when it comes out lopsided.

Where it comes from and how you spot it

Skew almost always traces back to a join or group-by on a column where one value dominates. The engine sends every row with the same key to the same partition, so a popular key piles a mountain of rows onto one task. Two patterns cover most cases: a null or placeholder customer_id shared by millions of rows, which cannot even match the other side of a join, and one giant tenant that sends a hundred times more events than every other customer combined.

Say a Spark job aggregates events by tenant_id. Spark shuffles into 200 partitions and runs 200 tasks. 199 finish in seconds; one holds the enterprise tenant's hundred million rows and runs for an hour. The stage looks nearly done at 199 of 200, but runs at the speed of that last straggler, which is why you watch the slowest task, not the average. More workers do not help: they idle while one task grinds through one partition.

How to fix it

The right fix depends on the cause, and you often combine two.

• Filter the junk keys first. If millions of rows carry a null customer_id that can never match, drop them or route them aside before the join. This is the cheapest fix and often the whole problem.

• Broadcast the small side. When one table in the join is small, broadcast a copy to every worker so the big table never has to shuffle. Spark does this automatically below a size threshold.

• Salt the hot key. Add a random suffix, say 0 to 99, to the dominant key so its rows scatter across partitions, then aggregate in two stages. Salting is manual, but it is the standard fix for a group-by that one value dominates.

What Spark's adaptive query execution does and does not do

Spark's adaptive query execution (AQE), on by default since Spark 3.2, re-plans a query at runtime from the partition sizes it measures after a shuffle. One of its rules handles skewed joins: it splits an oversized partition into smaller tasks and replicates the matching side, so a sort-merge join no longer stalls on one giant partition.

It is not a blanket cure. AQE only splits a partition it judges skewed, larger than five times the median size and larger than 256 MB by default, so a smaller imbalance is left alone. It handles joins, so a skewed group-by or a data pipeline feeding a data warehouse still needs salting or a better partition key, and it will not filter your null keys. Coverage is narrower for some join types too, such as a left outer join where only the left side can be split.