Batch processing

What is batch processing?

Batch processing collects a set of work and runs it as one bounded job, on a schedule or a trigger, instead of handling each record the moment it arrives. A job wakes up at 03:00, reads every new order from the day, calculates margins, and writes the results so the morning dashboard is ready before anyone logs in.

The defining trait is the bounded set: when the job starts, it knows exactly which rows it has to process. Microsoft describes batch as suited to workloads that do not need immediate access to insights, where the time between data arriving and a result appearing is minutes or hours rather than seconds.

Think of a mailroom that sorts the whole day's post in one pass, rather than walking to the table every time a single envelope drops through the slot. Most business reporting runs this way.

How a batch job runs

Every batch job has the same four moving parts.

1. A trigger. A schedule such as nightly at 03:00 or hourly on the hour, or an event such as a file landing in storage. A workflow engine usually owns this step, firing the job and holding back anything that depends on it until the job succeeds.

2. A selection. The job decides which records are in scope. It can reprocess everything, or take only the rows changed since the last run, which is the incremental load pattern. Incremental keeps the set small and the run fast.

3. The processing. The job reads the source, filters, joins, aggregates, and writes the output to new tables or files. The shape is the same whether it runs as an ETL step, an ELT transformation inside the warehouse, or an Apache Spark job across a cluster.

4. A result and a log. The job records what it wrote and which records failed, so a problem points you at the bad rows, not the whole run.

Cloud batch services put concrete names on this. Azure Batch runs a job as a set of tasks on a pool of machines it creates for the run; AWS Batch submits jobs to a queue and provisions compute to match. Both release the compute when the work is done rather than keep a cluster idle.

Why batch is still the default for reporting

Real-time gets the attention, but batch quietly runs most of the numbers a business looks at. Three properties keep it there.

Simpler failure handling
A batch run is a clean unit. If it fails halfway, you fix the cause and run it again. Because the job reprocesses the whole set it was given, data that arrived late for an earlier period is picked up and the previous result is corrected on the next run, with no special code. A streaming job has to carry state and reason about out-of-order events to get the same correction.

Cheaper compute
You schedule the work for quiet hours, size the machines for that one run, and switch them off when it finishes. There is no always-on cluster waiting for the next event. Azure Batch and AWS Batch both lean on this, down to discounted spare-capacity machines for work that can tolerate a restart.

Easy reprocessing and backfills
When a definition changes, you edit the transformation in your data pipeline and run it again over history. A backfill is the same job pointed at an older date range. Rebuilding two years of a report becomes a scheduling decision, not a rescue operation.

None of this needs a clever job. It needs each run to be idempotent, so replaying it updates the same rows instead of creating duplicates. That property is called idempotence, and it makes a rerun or a backfill safe to repeat.

The batch window

Every scheduled job has a slot it is expected to finish inside. The nightly load has to be done before staff arrive and before the reports refresh. A job that feeds another job has to finish before that one starts. That slot is the batch window.

Trouble starts when a job outgrows its window. Volumes grow, the nightly run that used to take two hours now takes five, and it is still writing when the business day begins. Three things tend to happen, none of them good: the morning dashboard shows yesterday's stale numbers, the next scheduled run starts before this one has finished and the two collide, or a hard time limit stops the job outright. Azure Batch exposes this last case directly as a maximum wallclock time: pass it and the job and all its tasks are terminated.

The usual fixes are unglamorous and they work. Switch a full reload to an incremental load so each run touches less data. Partition the data by day so the job reads only the new partition. Give the run more machines. Or move the window to a quieter hour. Reaching for streaming here is usually the wrong reflex, because the job is slow, not late by design.

Batch versus streaming, honestly

Streaming data processing handles each event within seconds or less, which batch cannot match. That speed has a bill attached: an always-on system, and state to manage for events that arrive late or out of order. Batch trades that latency for simpler logic, results that reflect the full set, and a lower compute cost.

The engines have converged, which softens the choice. Apache Spark runs the same transformation as a scheduled batch job or as a continuous stream, so choosing batch now is not a dead end you must escape later.

The honest question is rarely batch or streaming. It is how fresh this number actually needs to be. Real-time is often asserted rather than needed. A monthly margin report does not change if it is an hour old. Fraud detection during a payment genuinely needs the event now, and that is a real streaming case. Ask what decision changes if the figure is an hour stale. If the answer is nothing, batch is the cheaper and calmer choice.