Apache Iceberg

What is Apache Iceberg?

Apache Iceberg is an open table format for large analytical datasets. It is used on top of object storage such as S3, Azure Data Lake Storage, Google Cloud Storage, or OneLake, where the actual data often lives as Parquet files.

Plain Parquet files are excellent for analytics, but a folder of files is not a database table. It does not know which files belong to the current version, how to handle two writers at the same time, or how to safely rename a column. Iceberg adds that missing table layer. It tracks schemas, partitions, snapshots, manifests, and the current table metadata so multiple engines can treat the files as one consistent table.

The result is a lakehouse table that can support time travel, schema evolution, partition evolution, row-level changes, and safe concurrent reads and writes without moving the data into a proprietary warehouse format.

How Iceberg works

An Iceberg table is a set of data files plus a metadata structure that points to the right files for each version of the table.

Data files
The actual rows live in files, usually Parquet, though Iceberg can also use formats such as ORC and Avro.

Manifest files
Manifests list data files and store useful statistics such as partition values, record counts, and column bounds. Query engines use this metadata to skip files that cannot match a filter.

Manifest lists
A snapshot points to a manifest list. That list points to the manifests that make up the table state for that snapshot.

Table metadata
The table metadata file tracks the current schema, partition specification, properties, and snapshots. A commit creates a new metadata file and swaps the table pointer to it. Readers keep using the snapshot they loaded, so they do not see half-finished writes.

That design gives Iceberg a clean version history. If a bad load lands, you can query an older snapshot. If two writers try to commit at once, Iceberg can detect the conflict instead of silently losing one writer's work.

Why Iceberg improved older lake tables

Before open table formats, many data lakes used Hive-style tables: a metastore plus folders partitioned by values such as date or region. That worked, but it became fragile at scale.

Schema changes were risky
Iceberg tracks columns by ID, not just by name or position. That makes operations such as adding, dropping, renaming, and reordering columns much safer than relying on file position.

Partitions can evolve
In a Hive-style table, changing from daily partitions to monthly partitions is often a migration project. In Iceberg, the partition specification can change for new data while older data remains readable.

Query planning is more efficient
Because Iceberg keeps file-level metadata in manifests, engines can skip unnecessary files without listing every folder in object storage.

Writers can be concurrent
Iceberg uses optimistic concurrency around metadata commits. Writers prepare changes, then commit only if the table has not changed in a conflicting way.

Where Iceberg fits in a lakehouse

Iceberg is strongest when a table needs to be shared across engines and vendors. Spark, Flink, Trino, Presto, Snowflake, Dremio, AWS Athena, and other engines can all work with Iceberg in different ways. That engine-neutral posture is one of the reasons organisations choose it.

In Microsoft Fabric, Delta Lake is still the native default for lakehouse tables. Iceberg matters when data already exists in Iceberg elsewhere, or when an organisation wants interoperability with tools outside the Microsoft stack. OneLake supports working with Delta and Iceberg tables through metadata virtualisation, so Iceberg tables can be exposed to Fabric workloads without copying the underlying Parquet files.

The design decision is usually less about one feature and more about platform strategy: are you standardising on Databricks or Fabric, or do several engines need equal access to the same open table?

Iceberg versus Delta Lake and Hudi

Delta Lake
Delta is the obvious choice inside Databricks and Microsoft Fabric. It is tightly integrated with Spark and has a large operational ecosystem.

Apache Iceberg
Iceberg is the stronger candidate when neutrality and multi-engine access matter. Its snapshot and manifest model is designed for very large tables and for changing partition strategies over time.

Apache Hudi
Hudi is often chosen for update-heavy and CDC-heavy workloads, especially when upserts and incremental queries are the centre of the design.

All three formats can store data as Parquet and all three bring database-like guarantees to object storage. The right choice depends on the platform, the writer engines, the reader engines, and the operational skills around the table.

What to watch out for with Iceberg

The catalog is a real architecture choice
Iceberg needs a catalog to track table locations and metadata. Hive Metastore, AWS Glue, REST catalogs, Nessie, Polaris, and vendor catalogs each change what tools can read and write easily.

Small files do not disappear
Iceberg can plan queries efficiently, but thousands of tiny files still slow reads and metadata handling. Compaction remains part of operating the table.

Feature support differs by engine
One engine may support a newer Iceberg feature before another does. Check versions before relying on row-level deletes, newer spec features, or a specific catalog behaviour.

Schema evolution is not magic
Adding or renaming columns is safe. Changing meaning, incompatible data types, or business logic still needs review, tests, and downstream communication.