Streaming data

What is streaming data?

Streaming data is data that is processed continuously, event by event, as it arrives, instead of being collected and run through later on a schedule. The stream has no natural end. As long as orders are placed, sensors report, or payments clear, new records keep coming, so the processing never stops either.

The unit of a stream is an event: a small, timestamped record that something happened. "Customer 4212 paid 87 euros at 14:32." Put those events in the order they occurred and you have an event log, a running account of what happened and when. Apache Spark Structured Streaming captures the model: it treats a live stream as a table that is continuously appended, so you write an ordinary query and the engine keeps the answer current as rows arrive.

The counterpart is batch processing: records gathered into a fixed set and run through in one pass, usually overnight or hourly. Batch tells you what happened up to the last run; streaming tells you what is happening now, within seconds.

Event time versus processing time

Every event carries two clocks, and they rarely agree. Event time is the moment the event actually happened on the device that produced it. Processing time is the moment your system gets around to handling it. A payment made at 14:32 on a phone with no signal might not reach your pipeline until 14:50.

The gap matters because events arrive out of order. Count sales into five-minute buckets by processing time and that late payment lands in the 14:50 bucket, overstating the evening and understating the afternoon. Group by event time and it falls into the 14:30 bucket where it belongs, no matter when it turned up.

The snag is that you cannot wait forever for stragglers. Deciding how long to hold a bucket open before you treat its count as final is the job of a watermark, a concept worth reading up on in its own right.

Windowing: tumbling, sliding and session

Because a stream never ends, most questions about it are really questions about a slice of time. A window is that slice, and three shapes cover almost everything.

A tumbling window has a fixed size and does not overlap, so every event falls into exactly one window. "Orders per five minutes" is a tumbling window: 14:00 to 14:05, then 14:05 to 14:10, and so on.

A sliding window also has a fixed length but starts again on a shorter interval, so the windows overlap and one event can sit in several at once. "The order count over the last five minutes, refreshed every minute" is a sliding window; each event is counted in up to five of them.

A session window has no fixed size. It groups events by bursts of activity and closes after a set gap of silence. One shopper's clicks form a session that ends once they have been idle for, say, thirty minutes.

Delivery guarantees: at most once, at least once, effectively once

Networks drop messages and machines restart, so a streaming system has to promise something about how often each event is handled. There are three levels.

• At most once. Each event is handled zero or one time. Nothing is duplicated, but an event can be lost if something fails at the wrong moment. Acceptable for a rough metric where the odd gap does no harm.

• At least once. Each event is handled one or more times. Nothing is lost, but an event can be processed twice. This is the common default. Microsoft Fabric's Eventstream, for instance, documents at-least-once delivery.

• Effectively once. An event may still arrive twice, but the end result is recorded only once. This is what people usually mean when they say "exactly-once," and it is the honest way to describe it.

True exactly-once is real but bounded. Apache Kafka's idempotent producer strengthens delivery from at-least-once to exactly-once so producer retries no longer create duplicates, and Spark can reach end-to-end exactly-once when the source is replayable, the engine uses checkpointing, and the sink is idempotent. Step outside that boundary, say by writing to an external system that takes no part in the guarantee, and you are back to at-least-once. The safe answer is to make your own processing idempotent, so handling an event twice ends the same as handling it once.

When streaming earns its cost

Most business questions do not need any of this. Monthly revenue, quarterly churn, last week's best sellers: a batch that refreshes overnight or hourly answers all of them, and a scheduled run is cheaper to build and to keep alive than an always-on stream. Streaming infrastructure runs day and night and needs watching, so reach for it only when a delay of minutes has a real cost.

Cases where it does:

• Fraud checks. A suspect payment can only be stopped in the moment. The same finding the next morning is too late to act on.

• Cross-channel stock. Sell the same item in a shop, a webshop and a marketplace, and you need the count updated per sale so you do not sell what is already gone.

• Machine monitoring. Vibration and temperature often signal a fault before a line stops. Whoever watches the stream can step in before the standstill, not read about it afterwards.

There is a middle ground. Change data capture turns an ordinary database into a near real-time feed by emitting every insert, update and delete as an event, which keeps your warehouse fresh without the weight of a full streaming platform. Start with batch, add change data capture when a nightly refresh is genuinely too slow, and keep true streaming for the cases where seconds count.