Text classification

What is text classification?

Text classification is putting a label from a fixed list onto a piece of text. Which queue a support mail belongs in. Which cost category an invoice line falls under. Whether a document is a contract or an invoice. Whether a message is spam.

The fixed list is what separates it from the rest of language work. A summariser can write anything it likes; a classifier picks one of six options you wrote down in advance. That constraint is the point, because the output is a value in a column, so you can filter on it, count it and hang a workflow off it.

Two shapes exist and you pick one before you start. In single-label classification every item gets exactly one label and the labels exclude each other: a mail goes to one queue. In multi-label classification an item can carry several at once, or none: a contract can be tagged both privacy and intellectual property. In spaCy the single-label component returns scores that sum to 1, while for the multi-label one, in its own documentation, "there is no particular guarantee about their sum". Azure makes you pick the project type when you create the project.

Sentiment analysis is text classification with one particular label set, usually positive, negative and neutral. Data classification in the governance sense, where a document is public, internal or confidential, is a different exercise: that label comes from policy and pattern matching rather than from a model that learned your business from examples.

Three ways to build one

All three run in production today and they are not stages you pass through. Each wins on a different axis.

Rules. A lookup table or a list of patterns. Mail from the courier's noreply address goes to the delivery queue. An invoice line from your electricity supplier is energy. Rules cost nothing, they are exact, and anyone can read why a decision happened. They break the moment the wording varies. Start here anyway: in most SME data a handful of rules already covers a large share of the volume, and that share never needs a model.

A trained classifier. You label a few hundred examples, a model learns which word combinations go with which label, and it scores new text in milliseconds. That is ordinary supervised learning. Scikit-learn's own worked example gives the scale: four newsgroup categories, about two thousand training documents, a linear model that trains in hundredths of a second and lands around 75 to 78 percent accuracy on a deliberately hard test set. After training the running cost is near zero, which is why this option still wins at high volume.

A language model with the label list in the prompt. You send the text plus the labels, one sentence explaining each, and read the answer back. No training run, no corpus to build first. Anthropic's ticket-routing guidance puts it plainly: a pre-trained model "can effectively classify tickets with just a few dozen labeled examples". You pay per row and per token, and the behaviour can move when the provider ships a new model version.

The label set is the hard part

Teams spend three weeks on the model and one hour on the labels. That ratio is backwards. Three failure modes account for most disappointing classifiers, and all three are label problems.

Labels that overlap. Technical issue and delivery problem both fit a mail saying the machine has not worked since it arrived. No model settles that, because two colleagues cannot settle it either. Microsoft's guidance for its own classification service says the same from the other side: the more ambiguous your schema, the more labelled data you need to tell the classes apart, and when two classes keep getting mixed up you should consider merging them.

A catch-all that swallows the volume. Every label set grows an "other". Left undefined it expands quietly, because everything the model is unsure about goes there and nobody reads it. Once a third of the mail sits in "other" the classifier has stopped doing anything. Write a definition for it, decide what share is acceptable, and read what landed there every month.

Labels nobody can define in a sentence. Put the same fifty items in front of two colleagues separately. If they agree on forty, forty out of fifty is roughly the ceiling for any model built on their labels. One sentence per label with two examples and one counter-example is the actual work, and it is what gets skipped.

For a trained model Microsoft recommends around 50 labelled documents per class and warns that a class with fewer than 15 can drag accuracy down. A prompt needs far fewer examples to work, but you still want a labelled set that size, because without it you have nothing to measure against.

How you tell whether it works

Overall accuracy is close to useless here. With six queues where one holds 30 percent of the volume, a model that handles the big queue well and the small ones badly still reports a respectable number, and that number hides exactly the queues you were worried about. Read precision and recall per label instead. Precision on a label answers: of everything filed here, how much belonged here. Recall answers: of everything that belonged here, how much did it find.

label                 precision  recall  support
invoicing                  0.95    0.96      180
delivery                   0.91    0.94      150
technical issue            0.79    0.79      108
quote request              0.88    0.71       84
return or complaint        0.90    0.87       60
other                      0.33    0.56       18

Overall accuracy on that run is 87 percent, which sounds fine. Two rows spoil it. Quote request has recall 0.71, so three out of ten quote requests never reached the sales queue, and that is the mistake in this list that costs money. Technical issue has precision 0.79, so one in five mails sent there did not belong. Those are the same fact seen from two sides: a confusion matrix shows most of the missing quote requests sitting in the technical issue column, because customers describe the problem before they ask what a replacement costs. Microsoft's advice for that reading is either to merge the two classes or to label more examples that sit between them.

Class imbalance is why "other" scores so badly: 18 cases is too little to learn from and too little for the score to be stable. And a confusion matrix only exists for single-label work. Azure states plainly that it is not available for multi-label projects, because a document with two correct labels and one prediction is not a square you can put in a grid.

Two worked examples

Routing 3,000 support mails a month into six queues

A wholesaler gets about 3,000 mails a month on one address. Sorted by hand the split runs 900 invoicing, 750 delivery, 540 technical, 420 quote requests, 300 returns and complaints, 90 genuinely other. Someone opens each mail and drags it to a folder, roughly 30 seconds each, so 25 hours a month.

Running that through a prompt costs almost nothing. Each call carries the mail plus the label list and instructions, call it 500 input tokens, because the labels travel with every row rather than once. That is 1.5 million input tokens a month. At Claude Haiku 4.5's price on 4 September 2026, 1 dollar per million input tokens and 5 dollars per million output tokens, with an answer of a label and one line of reasoning at about 40 tokens, the month costs about 1.50 dollar in and 60 cents out. That is two dollars, or one through the batch API at half price.

What you actually decide is where the automatic filing stops. At a threshold that lets through 78 percent of the volume, 2,340 mails file themselves and 660 go to a review queue at 20 seconds each, about three and a half hours. Of the 2,340, roughly four percent land in the wrong queue and get moved later at a couple of minutes each, so add three hours. Seven hours of handling instead of twenty-five, and the three hours of rework are the number to watch, not the two dollars.

A trained classifier gives you a probability you can threshold honestly. Ask a language model how sure it is and you get a number it wrote, not a measured probability, so bind the decision to something sturdier: file it when the model names one label and quotes the sentence it decided on, queue it when the answer is "other" or when the reasoning names two labels.

Classifying purchase invoice lines into cost categories

The same company books about 1,000 purchase invoice lines a month into eight cost categories. Before any model touches it, a supplier lookup handles 620 of them, because 23 suppliers always book to the same category and always will. The remaining 380 go to the classifier at roughly 300 input tokens a line including the eight label definitions, so 114,000 tokens a month, around 11 cents. You could run it for a year on the price of one lunch.

The per-class report is where the work sits. Energy scores near perfect because the wording is unmistakable. External services and maintenance mix constantly, because a maintenance contract that includes parts genuinely is both, and the fix is a written rule about which one wins, not a better model. Lines whose description is a bare article code cannot be classified from the text at all, so they go to the bookkeeper by design rather than by failure.

A trained classifier versus a prompt, on the cost of changing the label set

Take one concrete change: the wholesaler starts selling service subscriptions and needs a seventh queue.

With a trained classifier you first find about fifty examples of the new class, which means digging through mails already labelled invoicing or other. Then you re-check the existing labels, because a share of last year's invoicing mails are now subscription mails. Then you retrain, re-evaluate every class rather than only the new one, and redeploy. The scores of the neighbouring classes will have moved, so somebody has to look at those too. That is days of work, spread over people who have other jobs. Anthropic's guidance describes it in one line: once a traditional machine learning approach is established, changing it is "a laborious and data-intensive undertaking".

With a prompt you add a line to the label list, add two examples, rerun your test set of 600 mails and compare the per-class numbers against the previous run. That is an afternoon.

That flexibility has a price. The trained model is frozen, so the same mail gets the same label next year. The prompt is not, because the model behind it gets versioned and retired and a new version can decide borderline cases differently without you touching anything. Pin the model version, keep the test set, rerun it when you move.

What to watch out for with text classification

Drift arrives as a new product line, not as a warning. The classifier has never seen the thing you started selling in March, so those mails go to "other" or to a plausible wrong queue. Watch the weekly share of "other" and the share of low-confidence answers. A jump in either shows up months before anyone complains.

Measure on live traffic, not on a balanced sample. A test set with equal numbers per class is easier to build and flatters the model. Your real mail is 30 percent invoicing and 3 percent other, and that skew is what the small classes suffer from.

The threshold is a business decision. Filing more automatically means fewer reviews and more rework. Cost out both sides once, in minutes per case, and the threshold picks itself.

Store the label with its evidence. Keep the label, the confidence and the model version next to the record, so a question six months later has an answer.

Do not silently relabel history. Old records keep their old label unless you rerun them, and a report comparing this quarter to last is then comparing two different label sets.