Small files problem

What is the small files problem?

The small files problem is what happens when a data lake or lakehouse table ends up spread across thousands of tiny files instead of a smaller number of larger ones. Each file might hold only a few kilobytes or megabytes, but the query engine still has to find, list, open and schedule every one.

It has little to do with how much data you store. Past a point, the engine spends more time on file bookkeeping than on your rows, so even a small table reads slowly.

Why so many small files slow reads down

A read that looks trivial still pays four kinds of overhead.

• Opening the file. Before any rows come out, the engine opens the file and reads its footer. Parquet keeps its column statistics and layout in that footer, so with tiny files the step repeats thousands of times for almost nothing.

• Listing files on object storage. On S3, ADLS or GCS there is no cheap folder to scan. Files are found through storage API calls, so thousands of objects mean thousands of metadata requests.

• One task per file. Distributed engines assign about one task per file, so thousands of files become thousands of short tasks whose scheduling cost dwarfs the few rows each reads.

• Metadata pressure. The table tracks every file it owns, so a Delta Lake transaction log or an Apache Iceberg manifest grows with the file count. On classic HDFS the NameNode holds about 150 bytes per file and block in memory, so ten million files cost roughly three gigabytes.

A worked example

Say a streaming job writes one micro-batch a minute into a lakehouse table, each batch as its own file. That is 1,440 files a day, so after a month the table holds around 43,000 files. If each file averages 50 KB, the whole table is only about two gigabytes.

Compacted into 128 MB files, that same two gigabytes fits in roughly sixteen files. A query that should touch sixteen instead plans and opens more than forty thousand. Nothing about the data changed, only the layout.

Where small files come from

• Streaming and micro-batches. Every short batch commits its own file, even when it carries only a few records.

• Frequent small appends. Incremental loads that add a little data on a schedule pile up file by file, and without maintenance they never merge on their own.

• Over-partitioning. Splitting a table on too many columns, or on a high-cardinality one, scatters the rows across folders that each hold almost nothing. This is the failure mode partitioning warns about.

Keeping small files in check

The usual remedy is table compaction: rewriting many small files into fewer, larger ones. Delta Lake does this with its OPTIMIZE command, and Apache Iceberg has an equivalent rewrite action, both aiming for files in the 128 MB to 1 GB range. That side is covered under table compaction.

Prevention helps as much as cleanup: batch writes instead of emitting a file per handful of rows, enable write-time compaction, and partition with restraint. Larger files also give Parquet more room for its statistics and compression.