Data quality

What is data quality?

Data quality is the degree to which data is fit for the purpose you use it for. An address list has good quality for parcel delivery if parcels arrive. The same list may be too weak for invoicing if VAT numbers or legal names are missing.

Quality depends on both the data and the use case. The same table can be good enough for a marketing segment and unusable for a finance process.

Think of a sloppy stock count. The mistake happens in the warehouse, but the pain appears at checkout when the webshop sells an item that is no longer on the shelf. Data behaves the same way: errors often start at entry but show up much later in reporting, automation, or customer contact.

The common six dimensions

Good data is too vague to manage, so teams split quality into measurable dimensions. A common DAMA-style set uses six.

• Accuracy
  Does the value match reality? A customer who moved last year but still has the old address in the CRM has an accuracy problem.

• Completeness
  Are the fields you need filled in? An empty comments field may not matter. A missing VAT number for an active business customer can block invoicing.

• Consistency
  Do systems agree? The same customer name may appear differently in CRM and accounting, or revenue in the ERP may not match the dashboard.

• Timeliness
  Is the data available when the process needs it? Yesterday evening's stock position may be fine for a monthly report and useless for real-time ecommerce.

• Uniqueness
  Does each real-world customer, product, supplier, or order appear once? Duplicate records are one of the fastest ways to break reporting and operations.

• Validity
  Does the value follow the expected format or domain? An email without an at sign is invalid. A date stored as loose text is invalid. Valid is not the same as accurate: blue can be a valid eye colour even if the person's eyes are brown.

Tools use slightly different labels. Microsoft Purview data quality, for example, also talks about measures such as freshness and conformity. The exact vocabulary matters less than making the checks measurable.

How good does data need to be?

Perfect data is not a practical goal. Every extra percentage point costs work: stricter entry rules, more cleanup, more validation, more exception handling. At some point the cost of perfection is higher than the cost of the remaining errors.

The better question is: what must be true for this decision or process to work?

Invoice data needs to be very accurate because errors create credit notes, disputes, and cash-flow problems. Marketing segmentation can tolerate more uncertainty. A sales forecast may be useful even when some opportunity probabilities are rough estimates.

That is why quality rules belong at dataset and field level. The active customer VAT number must be present and valid. Order_id must be unique. Product category must use the approved list. Those are rules you can test and improve.

Errors travel downstream

Bad data often does not hurt where it is created. A salesperson creates a duplicate customer and moves on. The deal is logged, so the immediate job feels done.

The duplicate then travels. The integration copies it to accounting. The warehouse loads it. The dashboard counts two customers. A campaign sends two emails. Every technical step may be green, yet the business result is wrong.

By the time the error appears in a report, the person who sees it is often not the person who caused it. The investigation then takes longer than the fix: source issue, connector issue, transformation issue, model issue, or report issue?

How do you improve data quality?

Measure with rules
Start with rules that can run automatically: every order number is unique, every active customer has a valid VAT number, every order references an existing customer, every daily load has a plausible row count. Data testing tools such as dbt, GX, Soda, and platform-native rules can show exactly which rows fail.

Assign ownership
If nobody owns the CRM, ERP, product list, or finance model, quality issues remain tickets without an owner. Name who receives issues, who decides priorities, and who schedules cleanup.

Fix the source where possible
A report filter that hides duplicates fixes one report and leaves every other consumer exposed. A source-system validation rule or corrected master record fixes the problem for everyone.

Keep monitoring
Quality is not a spring-cleaning exercise. New records arrive every day, and process changes create new error types. Track trends over time.

Quality, governance, and observability

Data quality, data governance, and data observability overlap, but they are not interchangeable.

Data governance defines who owns data, which definitions apply, who may access it, and which quality expectations matter.

Data quality is the degree to which the data actually meets those expectations.

Data observability monitors production data health: freshness, volume, schema, distribution, and lineage. It catches incidents and surprises, including problems nobody wrote as explicit quality rules.

For core records such as customers, suppliers, employees, and products, master data management is often part of the answer. MDM tries to keep one trusted master record across systems.

What bad data costs

Duplicate mailings
A customer appears three times and receives the same promotion three times, or receives a welcome email after ten years as a customer.

Wrong stock
The webshop sells a product that is no longer in the warehouse, or purchasing orders extra stock because the system shows a shortage that does not exist.

Invoices to the wrong address
A moved customer receives paperwork late, payment terms are missed, and the relationship suffers even though the customer intended to pay.

Reports nobody trusts
One visibly wrong number in a management meeting can damage trust in the entire reporting setup. People retreat to private spreadsheets, and rebuilding trust takes longer than fixing the original error.

What to watch out for

Do not chase a universal score
A single percentage can hide the field that actually matters. Track the rules that protect real decisions and processes.

IT cannot solve it alone
IT can build checks and integrations. The business must decide which value is correct and which trade-offs are acceptable.

Tools produce queues, not ownership
A quality platform can surface issues. It cannot decide who cleans them up or which rule matters most.

Fixing downstream symptoms creates debt
Every report-specific workaround is a small debt. Prefer source fixes and shared rules where the same problem affects several consumers.