Data ingestion

What is data ingestion?

Data ingestion is the process of collecting data from its sources and bringing it into a system where you can store, process or analyse it. The sources are wherever the data already lives: an application database, a SaaS tool with an API, log files, sensors, a message queue, a flat file dropped on a server. The destination is usually a data lake, data warehouse or lakehouse.

It is the first step of almost any data pipeline. Nothing downstream, no transformation, no report, no model, happens until the data has been ingested. If ingestion is late or incomplete, everything built on top inherits the problem.

Ingestion is only the moving and landing of data. The cleaning and reshaping is a separate job. The line matters because how and when you clean depends on whether you ingest in batches or as a continuous stream.

Batch versus streaming ingestion

There are two broad ways to bring data in, and the choice drives most of the design.

Batch ingestion collects data over a period and loads it in one go. A nightly job pulls the day's orders at 2am. A file lands every hour and gets picked up. Batch is simpler to build and run, and cheaper for the same volume, so it covers most reporting needs. The trade-off is freshness: the warehouse is only as current as the last run.

Streaming ingestion brings data in event by event, as it is produced, with a delay of seconds or less. A payment, a click, a sensor reading flows in the moment it happens. This is what you need for live dashboards, fraud checks or alerting. The cost is complexity: you are running an always-on system that has to handle bursts, retries and out-of-order events.

A rough guide: if a report that is a few hours old is fine, batch is usually the right and cheaper answer. Reach for streaming only when the value of the data drops fast with age.

Where ingestion sits in the pipeline

Ingestion is the E in ETL and ELT, the extract-and-load part before any transformation. The two patterns differ in when the cleaning happens.

In classic ETL, data is transformed on the way in, so it arrives already shaped. In the more modern ELT approach, raw data is ingested first and transformed later inside the warehouse, using its compute to do the work. ELT has become common because cloud warehouses and lakehouses are cheap to store into and fast enough to transform in place. Either way, ingestion is the step that gets the raw data landed.

How change data capture fits

Copying an entire source table on every run is wasteful once the table is large. Change data capture, or CDC, is the technique that ingests only what changed: the inserts, updates and deletes since the last read, usually by reading the source database's transaction log.

CDC is what makes near-real-time ingestion from an operational database practical. Instead of a nightly full copy of a million-row table, you stream the few thousand rows that actually changed. It is a common bridge between a live application database and an analytical system, and tools such as Debezium are built around it.

What to watch out for with data ingestion

Schema drift breaks things quietly. When a source adds, renames or retypes a column, an ingestion job can fail or, worse, keep running and load wrong data. Watch source schemas and decide up front how changes are handled.

Idempotence and duplicates. A retried or overlapping load can bring the same records in twice. Ingestion needs a way to load the same batch again without creating duplicates, often through an upsert on a stable key.

API limits on SaaS sources. Pulling from a SaaS tool means living with its rate limits and paging rules. A naive full pull can get throttled or time out. Plan for incremental loads and backoff.

Freshness expectations. Users assume the dashboard is current. If it refreshes once a night, say so. Mismatched expectations about how fresh the data is cause more distrust than the delay itself.