Data labelling and ground truth

What is data labelling and ground truth?

A label is the correct answer attached to one example. This invoice line is a transport cost. This email is a complaint. This box in the photo is a pallet. Data labelling is the work of producing those answers, one example at a time, so that a model can learn from them or be measured against them.

Ground truth is the set of labels you have decided to treat as correct. It is the answer key. When a classifier says "transport cost" and the ground truth says "packaging", the classifier is counted wrong. When an AI agent drafts a reply to a customer mail and the ground truth says the mail was a complaint, you check whether the agent treated it as one.

Supervised learning assumes labelled data exists. Evals assume a set of expected answers exists. Neither of them says where those answers come from, and in practice that is where most of the time goes. The answer key is written by people, under time pressure, with a definition of "correct" that someone had to invent.

Where labels come from

There are five common sources, and most real projects mix at least two.

People with domain knowledge. The bookkeeper reads the invoice line and picks the category. The service manager reads the ticket and says whether it is a complaint or a question. Google's machine learning course puts the trade-off plainly: human raters handle tasks a model finds hard and force you to write clear criteria, but you pay them, and because people make mistakes several of them may have to look at the same example.

Decisions your systems already recorded. Every booked invoice in your ERP carries the general ledger account a person chose. Every closed helpdesk ticket has the category an agent picked. Every quote has a won or lost flag. Those are labels produced by people during years of normal work, and they cost nothing to extract. The catch is that they record what people did, which is not always what the procedure says they should have done.

Weak rules. Instead of labelling each example, you write small rules that label many at once: an invoice from a courier company is probably a transport cost, a ticket containing "refund" is probably a complaint. The Snorkel team that started at Stanford calls this weak supervision: noisier sources of supervision, combined to build much larger training sets much faster than labelling one by one. The rules contradict each other, and the method learns from where they agree and where they clash how much to trust each one.

Model pre-labelling with human correction. A model, often a general-purpose LLM, proposes a label for every example and a person accepts or corrects it. Label Studio, an open source labelling tool, supports this directly: you import the model's predictions and they appear as pre-annotations that the annotator can copy and fix. Correcting is faster than labelling from scratch, with one catch covered below.

Synthetic examples, used with care. When a class is rare, a model can generate plausible examples of it. Useful for filling gaps, dangerous as ground truth, because a generated example carries the label the generator assumed rather than a label a person confirmed.

Why quality beats quantity

A wrong label is worse than a missing one. A missing label leaves a gap; a wrong label teaches the model the wrong answer and then scores it as right when it repeats the mistake. Four habits keep the labels honest.

Guidelines with examples. Write down what each label means and show two or three real cases for each, including the ones that look like a neighbouring label. "Transport cost" needs a sentence about the delivery charge printed on a goods invoice, because that line is where two labellers will split.

Two labellers on a sample, and a number for their agreement. Have two people label the same subset independently and count how often they agree. Plain percentage agreement flatters you, because with few categories two people who guess will agree part of the time by chance. Cohen's kappa corrects for that: it compares the observed agreement with the agreement chance alone would produce, and runs from 1 for perfect agreement down to 0 for agreement no better than chance. The scale most people use reads 0.61 to 0.80 as substantial and above 0.80 as almost perfect, though those cut-offs are a convention, not a finding. Low agreement is a message about the guideline, not about the labellers: the definition is not sharp enough yet.

Edge cases decided once. Every disagreement that reaches an adjudicator becomes a rule in the guideline. The point is that the same question never gets decided twice by two different people in two different ways.

A gold set the labellers do not see. Keep a small set of examples labelled by your most trusted expert and use it to check everybody else, including any model that pre-labels. Google's guidance is to double-check your raters by labelling some examples yourself and comparing, and Anthropic's engineering team gives the test for a good eval case: two domain experts, working independently, would reach the same verdict.

How many labels do you need

The honest answer is "fewer than you fear for evaluation, more than you hope for training". Rounded ranges, from vendor guidance and from our own practice:

• For an eval set. Anthropic's engineering guidance from January 2026 says 20 to 50 tasks drawn from real failures is a good start for an agent, and its evaluation guide gives examples like 200 articles with reference summaries or 100 customer enquiries, with the advice to prefer more cases graded automatically over a handful graded by hand. Fifty to two hundred real cases that a domain expert has signed off is the range we aim for before any AI project goes live.

• For fine-tuning an LLM. OpenAI's fine-tuning documentation sets the minimum at 10 examples and recommends starting with 50 well-crafted ones, adding more only if those 50 move the result. Thousands is the territory of changing how a model writes or reasons, not of teaching it your categories.

• For a classic classifier. It depends on the number of classes and how imbalanced they are. A rule of thumb from practice: enough examples that your rarest class still has a few hundred, which for a six-category invoice model with one rare class can mean several thousand lines in total.

In every case the constraint is the rare and ambiguous examples, never the easy ones. Two hundred typical invoice lines teach a model less than fifty typical ones plus fifty of the awkward ones your bookkeeper had to think about.

A worked example: 200 invoice lines

Say you want an assistant that classifies purchase invoice lines into six categories: goods for resale, transport, packaging, external services, energy, and other. Before you touch a model you build the ground truth.

1. Sample. Pull 200 lines from the last twelve months, not at random. You take the top suppliers proportionally, then deliberately add lines from small suppliers, credit notes, lines with a zero amount and lines whose description is a bare article code. Roughly 60 percent typical, 40 percent chosen because they look hard.

2. Two labellers. The bookkeeper and the purchasing lead each label all 200 lines on their own, using a one-page guideline with two examples per category.

3. Count the disagreements. They agree on 172 lines and disagree on 28, so 86 percent agreement. With six categories and this spread, chance agreement is around a quarter, so kappa lands near 0.8. Good, and the 28 lines are the interesting part.

4. Adjudicate. The finance manager rules on each of the 28. Twenty are simple slips. Eight are real gaps in the guideline: the delivery charge on a goods invoice, a supplier that sells both packaging and goods, a maintenance contract that includes parts. Each of those becomes a sentence in the guideline, decided once.

5. The gold set. The 200 adjudicated lines are the ground truth. Forty of them, mostly the hard ones, are set aside as a gold set nobody labels again. Any new labeller and any model is scored against those forty first.

Total effort: about a day and a half of two people's time plus two hours for the finance manager. That is the cheapest part of the project and the part that decides whether the numbers you see later mean anything.

Ground truth versus model output

The question that decides the comparison is which one you trust when they disagree. The reflex is to trust the ground truth, because that is what it is for. The better reflex is to check the label first.

A model that scores 92 percent against your ground truth is wrong on 8 percent of cases, or the ground truth is, or both. In our experience a share of the "errors" a decent model makes on a hand-labelled set turn out, on inspection, to be label mistakes: the bookkeeper booked it to the wrong account in a hurry, the guideline changed halfway through the year, two labellers never agreed and one of them won.

So the working rule is: when the model and the label disagree, a person looks at that case before anyone counts it. If the label was wrong, fix the label, and the ground truth gets better. If the model was wrong, keep the case, because it is exactly the kind you want in the gold set. Ground truth is the thing you measure against and the thing you keep correcting, at the same time.

What to watch out for with data labelling

Ground truth is not the truth. It is a set of decisions made by particular people under a particular definition on a particular date. Business definitions change: last year "external services" included software subscriptions and this year they are a category of their own. Labels from before the change are now wrong, and a model trained on them will keep booking subscriptions the old way with full confidence. Date your labels and re-check the gold set when a definition moves.

Labels from the ERP encode yesterday's mistakes. If a former colleague booked every courier invoice to "other" for three years, that is what the extracted labels say. Sample and check historical labels before you trust them, exactly as you would a new labeller.

Pre-labelling anchors the corrector. When a model proposes a label, a tired person tends to accept it. Keep a share of examples that are labelled blind, without the model's suggestion, and compare the two streams. If the blind stream disagrees with the pre-labelled stream more than the two labellers disagreed with each other, the model is steering the labels.

The cost is labour, and it is domain labour. The person who can label an invoice line correctly is the person who is already busy closing the month. Labelling services and crowd platforms are cheap per item and of little use for anything that needs your context. Plan the domain expert's hours into the project like any other cost, and spend their time on the hard cases while rules and pre-labelling handle the easy ones.

Tooling is the smallest problem. A spreadsheet with one column per label and a comment field works for a few hundred cases. Label Studio and similar open source tools handle images, documents and several annotators, and its paid edition adds agreement dashboards. Vendor labelling services exist for scale. Pick the tool after the guideline is written, never before.