Unstructured data

What is unstructured data?

Unstructured data is data that does not sit in a fixed shape of rows and columns. A signed contract, an email thread, a photo of a damaged pallet, a recorded support call, a scanned delivery note, a design file: none of them have fields you can query.

The word unstructured is slightly misleading, and it helps to know why. A contract has plenty of structure. It has clauses, parties, dates and amounts, and a lawyer reads it without difficulty. What it does not have is a structure a database understands. So the honest version of the term is: not structured in a way a machine can query directly.

For most organisations this is the larger half of what they own. Your ERP holds transactions, and everything explaining those transactions, the negotiation, the agreement, the complaint, the photo of the delivery, sits somewhere else entirely.

Structured, semi-structured, unstructured

Three categories are worth being able to separate, because they need different handling.

Structured data lives in a predefined schema. A table in your ERP, a spreadsheet with consistent columns, a row in your CRM. You know in advance what each field means and what type it is, and you can query it with SQL.

Semi-structured data carries its structure with it rather than in a schema. A JSON file, an XML message, a log line. There are fields and labels, but the shape can vary from record to record, and two messages of the same type need not have the same keys.

Unstructured data has no machine-readable fields at all. The meaning is in the content, and getting at it means interpreting the content itself.

The dividing line that matters in practice is not the format but the question of whether you can filter and aggregate without first interpreting the content. A PDF is therefore unstructured, even though it is a very orderly file.

Why AI reopened the question

Unstructured data is not new, and neither is the desire to use it. What changed is what you can do with it without a project per document type.

The old approach was extraction. You built a template that knew where the invoice number sat on a supplier's invoice, and it worked until that supplier redesigned their layout. Every document type meant a new template, and every layout change meant maintenance. That is why so many of these projects stalled after two document types.

Language models changed the economics of that. A model that reads a document and returns the fields you asked for does not need a template, and it copes with a layout it has never seen. Alongside that, embeddings made it possible to search by meaning rather than by keyword, so you can find the paragraph that answers a question even when it does not contain any of the words in the question.

Those two together are what put unstructured data back on the agenda. Not because the data changed, but because reading it stopped requiring a bespoke build each time.

What you can do with it now

Answer questions from documents. A retrieval setup over your policies, manuals or contracts, so people get an answer with a source reference instead of a folder to search through.

Turn documents into fields. Pulling supplier, date, amount and payment terms out of an invoice, or the notice period and the renewal date out of a contract, so it can go into a system as structured data.

Classify and route. Deciding what an incoming email or form is about and sending it to the right team, which often helps more than any clever answer, because it removes a queue.

Make recordings searchable. Transcribing calls and meetings, at which point a year of support conversations becomes something you can actually analyse.

Read photos. Damage reports, meter readings, condition on delivery. These are ordinary business processes that were never digitised because a photo was not data.

Notice that most of these end with unstructured data becoming structured data. That is usually the point. The value is rarely in the document itself, it is in the field you extracted from it and can now count, filter and report on.

What to watch out for with unstructured data

Volume is not the problem, findability is. Storage is cheap and the files are already there. The reason nobody uses them is that nobody can tell which version is current and which one was superseded.

Metadata is what makes it usable. Source, date, owner, audience, status. Without those, your retrieval cannot tell a draft from a signed version, or a policy from 2019 from the one that replaced it.

Permissions are harder here than in a database. Row level security on a table is a solved problem. A folder structure that grew over fifteen years is not, and a chat interface will happily surface a document that a person would never have found.

Personal data hides in it. A spreadsheet with a column called email is obvious. A CV in an old folder, a medical certificate in an email attachment, a name in a photo caption: those are the same obligations with none of the visibility.

Not everything deserves to be indexed. The instinct is to point the tooling at everything you have. A smaller, curated set of current documents gives better answers than a complete archive, because your outdated material stops competing with your current material.