Data lake

What is a data lake?

A data lake is a central place to store raw data in the format in which it arrives. That can mean ERP tables, JSON from APIs, log files, sensor data, PDFs, product photos, audio, or video.

The core idea is store first, structure later. A data warehouse usually applies a schema when data is loaded. That is schema-on-write. A data lake stores the files first and applies structure when a tool reads them. That is schema-on-read.

This is useful when you want to keep detailed source data before you know every future question. A report, Python script, Spark job, and machine learning model can read the same raw files in different ways.

Where did the term come from?

The term data lake is usually credited to James Dixon, then CTO of Pentaho. In a 2010 blog post, he contrasted a data mart with bottled water: cleaned, packaged, and structured for easy consumption. The data lake was the larger body of water in its natural state, filled from a source and available for users to examine, dive into, or sample.

The original context was Hadoop. Today, most data lakes run on cloud object storage such as Amazon S3, Azure Data Lake Storage, Google Cloud Storage, or OneLake in Microsoft Fabric. The storage technology changed; the idea stayed recognisable.

How a modern data lake works

Storage
A lake is usually built on object storage. Object storage is cheap, scalable, and good at holding many file types. You pay mostly for stored volume and access, not for a fixed database server.

File formats
Raw data often lands as CSV, JSON, XML, logs, images, or documents. Analytical layers usually convert structured data to Parquet because it is columnar, compressed, and efficient for query engines.

Query engines
Tools such as Apache Spark, SQL engines, notebooks, and lakehouse platforms read the files and apply the schema at read time. The same lake can serve engineers, data scientists, and analytics teams if the layers are organised well.

Zones
A healthy lake is not one giant folder. Many teams use zones such as raw, cleaned, and curated, or a medallion architecture with bronze, silver, and gold layers.

What do you use a data lake for?

1. Raw source archive. Keep every extract from source systems so you can reload or reprocess later without hitting the source again.

2. Machine learning. Models often need detailed history, logs, events, documents, or images that would be lost in a heavily summarised warehouse.

3. Logs, clickstreams, and sensor data. These are high-volume, semi-structured, and often hard to model upfront.

4. Documents and media. Contracts, invoices, photos, scans, and recordings do not fit naturally inside a relational warehouse but can live in a lake.

Data lake versus warehouse versus lakehouse

Data lake
Stores raw and varied data cheaply. It is flexible, but needs engineering, metadata, access control, and careful structure to be useful.

Data warehouse
Stores structured, cleaned, modelled data for analytics. It is usually better for business reporting because definitions, performance, and access are more controlled.

Lakehouse
Adds table formats and warehouse-like behaviour on top of lake storage. Delta Lake, Apache Iceberg, and Apache Hudi are examples of table formats used in lakehouse architectures.

In practice, the three can work together. A lake stores the raw archive. A warehouse or lakehouse curated layer serves reporting and analytics.

What is a data swamp?

A data swamp is a data lake without order. Files arrive with unclear names, no owners, no lineage, no retention rules, and no quality checks. Nobody knows which copy is current. Users stop trusting the lake and create new exports somewhere else.

The fix is metadata and governance. Use a data catalog, clear zones, naming conventions, access rules, retention, data quality checks, and owners. Cheap storage is not an excuse to store everything forever with no explanation.

When is a data lake overkill?

An SME with a few structured systems and ordinary reporting needs may not need a lake. A well-designed warehouse or reporting database can be simpler and cheaper to operate.

A lake starts to make sense when you have large volumes, many file types, raw historical archives, machine learning needs, or a lakehouse platform such as Microsoft Fabric where the lake is part of the standard architecture.

FAQ

Is a data lake a database?
Not usually. A database manages structured data with tables, indexes, transactions, and query optimisation. A data lake stores files and relies on engines and table formats to interpret them.

Does a data lake replace a data warehouse?
Usually no. The lake is good for raw and varied data. The warehouse is good for trusted business reporting. A lakehouse can bring the two closer together, but it still needs modelling and governance.

Can Power BI report directly on a data lake?
Yes, when the data is exposed as tables through a SQL endpoint, semantic model, Direct Lake, or another supported connector. Reporting directly on raw files is rarely the best user experience.

What is OneLake?
OneLake is Microsoft Fabric's built-in, organisation-wide logical data lake. Every Fabric tenant includes it, and Fabric workloads use it as the shared storage layer.