Data freshness measures how recent data is compared with the real-world event or source it represents. A data SLA makes that expectation explicit, such as a sales table that must never be more than one hour old.
Data freshness measures how recent your data is compared with the real-world event or source it represents. A table is fresh when it reflects what is currently happening closely enough for the decision at hand. It is stale when the world has moved on and the table has not caught up.
Freshness is not the same as correctness. A sales number can be perfectly correct and still be twelve hours old. For a monthly finance report, that may be fine. For stock availability in a webshop, it may be dangerous.
That is why freshness is always relative to use. The question is not is this data new? The question is is this data recent enough for this process?
Stale data looks trustworthy. The dashboard loads. The table has rows. The chart has labels. Nothing screams that the newest order, payment, shipment, or stock movement is missing.
Imagine an inventory report that is half a day behind. The report says there are 200 units in stock. The warehouse has 20. The purchasing decision feels data-driven, but it is based on an old snapshot.
The same thing happens in sales reporting. A pipeline dashboard that has not updated since yesterday can still shape today's meeting. A report does not have to be technically broken to be operationally misleading.
The simplest freshness check compares the newest timestamp in a table with the current time.
If the newest loaded_at timestamp is 08:00 and it is now 14:00, the table is six hours old. If the data SLA says the table must be no more than one hour old, the table is stale.
There are two useful timestamps to separate.
Event time
When did the business event happen at the source? For example, when was the order created?
Load time
When did that event become available in your warehouse, lakehouse, or reporting model?
The gap between event time and load time is pipeline latency. The age of the newest loaded row tells you whether the table has recently received new data at all. Both are useful, but the second is often the easiest first check.
A data SLA is an explicit freshness expectation. It turns this data should be up to date into a measurable rule.
Examples:
Sales orders must be less than one hour old during business hours.
Inventory must be less than fifteen minutes old for ecommerce availability.
Accounting transactions must be loaded by 07:00 every working day.
Marketing campaign spend may be up to 24 hours old.
Without a data SLA, stale data is a matter of opinion. With an SLA, it becomes an incident when the threshold is crossed.
A staleness alert warns when a dataset has not updated within the agreed limit. dbt's source freshness feature is a common example. You configure a timestamp field and set thresholds such as warn_after and error_after. dbt compares the most recent timestamp with the current time and reports whether the source is still fresh.
The two-threshold pattern is useful. A warning gives the team time to react before the data is officially late. An error says the SLA has been missed and consumers should no longer assume the data is current.
A practical rule: check more often than the strictest SLA. If data must never be more than one hour old, checking once per hour means you can discover the issue only after it is already late. Checking every fifteen or thirty minutes gives you room to respond.
Refresh frequency
A Power BI import model can only be as fresh as its refresh pattern. Microsoft currently limits scheduled refresh on shared capacity to eight times per day, and allows up to 48 scheduled refreshes per day on Premium per user, Premium, or Fabric capacity. For fresher reporting, you may need DirectQuery, Direct Lake, streaming, API-triggered refresh, or another architecture.
Incremental refresh
For large tables, refreshing everything may take too long. Incremental refresh reloads only recent partitions, which can make more frequent refresh practical.
Change Data Capture
CDC captures inserts, updates, and deletes from a source and moves them downstream quickly. It can reduce latency from hours to minutes or seconds, depending on the stack.
Data observability
Freshness is one of the core signals in data observability, alongside volume, schema, distribution, and lineage. Freshness checks tell you whether data arrived on time; the other signals help explain whether it arrived in the expected shape and quantity.
Pick critical tables. Start with the datasets used in daily operations, management reporting, billing, stock, or customer communication.
Find a reliable timestamp. Use a load timestamp, event timestamp, or warehouse metadata where supported.
Agree the threshold. Decide how old the data may be before it becomes risky.
Run a scheduled check. Compare the newest timestamp with the current time.
Alert the owner. Send the alert to someone who can fix or escalate the issue.
Tell consumers when freshness is broken. A stale dashboard should not silently look healthy.
You do not need an enterprise platform to start. A scheduled query and a Teams or email alert can catch the first class of silent failures.
Apache Hudi is an open table format for data lakes that makes Parquet files behave like transactional tables. It is strongest where data cha...
Read definitionApache Kafka is an open-source event streaming platform. Producers write events to topics, Kafka stores them as durable partitioned logs, an...
Read definitionAt-least-once delivery means every message should arrive one or more times. It protects against loss, but retries can create duplicates. Tha...
Read definitionA backfill is the rerun of a data pipeline over historical periods that were missed, wrong, or not yet processed. It is common when launchin...
Read definitionBatch processing collects a set of work and runs it as one bounded job on a schedule or a trigger, instead of handling each record the momen...
Read definition