Apache Hudi

What is Apache Hudi?

Apache Hudi is an open table format for data lakes. It sits on top of files in object storage, usually Parquet, and adds the table behaviour that plain files do not have on their own: transactional writes, row-level updates and deletes, schema changes, and a timeline of commits.

The simplest way to think about Hudi is this: it turns a folder of files into a table that knows which version is current and which rows changed. That matters when your source data does not just arrive as append-only batches. Orders change status, customers update details, invoices are cancelled, and GDPR deletion requests arrive. Hudi is built for those update-heavy patterns.

Hudi originally grew out of Uber's need to keep very large data lake tables fresh without constantly rewriting them. That history still shows. Compared with the other open table formats, Hudi puts extra emphasis on upserts, deletes, and incremental reads.

Copy-on-Write and Merge-on-Read

Hudi has two main table types. The choice decides where you pay the cost: during writes or during reads.

Copy-on-Write
In a Copy-on-Write table, Hudi rewrites the affected base files when records change. Reads stay simple and fast because a query only needs the current Parquet files. The trade-off is write cost: even a small update may require a larger file rewrite. This fits read-heavy reporting tables where writes happen in controlled batches.

Merge-on-Read
In a Merge-on-Read table, Hudi writes changes first to log files next to the base Parquet files. Writes are faster and can happen more often. A snapshot query then merges the base files and logs when it reads, unless you choose a read-optimised query that only reads compacted base files. Periodic compaction folds the logs back into new base files. This fits streaming ingestion, Change Data Capture, and tables with many small updates.

The practical rule is plain: choose Copy-on-Write when read performance is the priority, and Merge-on-Read when freshness and frequent updates matter more.

Why Hudi is useful for CDC and upserts

The feature that makes Hudi stand out is the upsert. Each record has a key. When new data arrives, Hudi can decide whether that key is new or already exists, then insert or update the row accordingly. You do not have to build the full merge logic yourself every time.

That is exactly what a CDC pipeline needs. A source database emits a stream of inserts, updates, and deletes. Hudi lands those changes in a lakehouse table while keeping the latest state queryable. Downstream jobs can also ask for only the changes since a previous commit. That is the incremental query pattern: process the delta, not the whole history.

For example, a webshop sends every order change to object storage. The same order may move from created to paid, shipped, returned, and refunded. With Hudi, those events can keep one current order row up to date while still letting the next layer pick up only the changes since yesterday's run.

Hudi versus Delta Lake and Iceberg

Delta Lake, Apache Iceberg, and Apache Hudi solve the same broad problem: they make a data lake behave more like a database table. They all add metadata and commit tracking around open files. The difference is emphasis.

Delta Lake is the natural default in Databricks and Microsoft Fabric. It is widely used, straightforward to understand, and deeply integrated with Spark-based platforms.

Apache Iceberg is strong when many different engines need to work on the same table. Its metadata model, snapshot handling, and partition evolution are designed for large multi-engine lakehouse setups.

Apache Hudi is usually the most deliberate choice. You pick it because the workload is update-heavy, because you need efficient deletes, because CDC is central, or because incremental reads are a first-class requirement.

The boundaries are not fixed. The formats keep borrowing ideas from each other, and interoperability projects make the choice less permanent than it used to be. Still, the design centre is different enough to matter.

When Hudi is a good fit

• CDC into a lakehouse. Source systems emit a constant stream of row changes and the target table must stay close to current.

• Upsert-heavy facts. Orders, subscriptions, tickets, payments, and inventory records change after first arrival.

• Incremental downstream jobs. The next layer should read only what changed since the last successful commit.

• Targeted deletes. Individual rows need to be removed without rewriting entire partitions by hand.

What to watch out for with Hudi

Merge-on-Read needs compaction
Fast writes are not free. If log files keep piling up, reads become slower because every query has more work to merge. Compaction must be scheduled, monitored, and tuned.

Small files still hurt
Frequent streaming writes can create many small files. Hudi has file sizing and clustering features, but they need sensible configuration.

The ecosystem is narrower
Hudi is mature, but fewer tools support it out of the box than Delta Lake or Iceberg. In a Microsoft-heavy stack, Delta is often the easier operational choice.

There are more decisions to make
Table type, record key, indexing, clustering, cleaning, and compaction all affect behaviour. Hudi gives control, but that control needs ownership.