Anomaly detection

What is anomaly detection?

Anomaly detection is the automatic search for data points, events, or patterns that do not fit normal behaviour. A payment that is ten times higher than usual, a machine that starts running hotter, a table that suddenly arrives half empty, or a product return rate that jumps above its baseline can all be anomalies.

Think of an experienced accountant scanning a stack of invoices and stopping at the one that feels wrong. Anomaly detection does the same kind of check at data scale. It does not prove fraud, failure, or bad data by itself. It gives you candidates worth investigating.

The hard part is not finding every unusual value. The hard part is deciding which unusual values matter. A busy Friday can be normal for a webshop, while the same number of orders at three in the morning may deserve a closer look.

What counts as an anomaly?

Point anomalies
A single value is strange on its own: a refund of 40,000 euros in a shop where the average order is 80 euros, or a temperature reading far outside the usual range.

Contextual anomalies
The value is only strange in context. One hundred orders per hour may be normal on a Monday afternoon and suspicious at night. Time, location, customer segment, product type, and season can all change what normal means.

Pattern anomalies
No single measurement looks impossible, but the combination does. A machine may show normal temperature and normal fuel use separately, while the relationship between those signals has started to drift. Fraud often works this way too: each action stays below a fixed threshold, but the full pattern stands out.

How anomaly detection works

Rules and thresholds
You define what is not allowed: a cold room above seven degrees, a payment above a fixed limit, a required field that is empty, or a row count below yesterday's minimum. Rules are simple to build and easy to explain.

Statistical baselines
The data sets its own boundary. You compare new values with a mean, median, spread, moving average, or expected time-series band. For time series, the model can account for trend and seasonality, so a December sales peak does not trigger the same alert as an unexpected midnight spike.

Machine learning
When you have labelled examples, such as confirmed fraud cases, you can train a model to recognise similar cases. That is supervised learning. When you do not have labels, unsupervised learning can learn the shape of normal data and flag points that fall in sparse or unusual regions. Common approaches include isolation forests, local density methods, clustering, PCA, autoencoders, and time-series models.

There is also a useful technical distinction. Outlier detection assumes the training data already contains unusual points and tries to find them. Novelty detection learns from mostly normal historical data and checks whether new observations still look like they belong.

Rules, statistics or machine learning?

Start with rules when normal behaviour is stable and the business rule is clear. A refrigerated truck must stay below a fixed temperature. A required VAT number cannot be blank. A negative quantity may be invalid. These checks are transparent and often catch the most common problems.

Use statistical baselines when normal moves. Website traffic, cloud spend, support tickets, and energy use often have daily, weekly, or seasonal rhythms. A fixed threshold either misses quiet-period spikes or floods the team during normal peaks. A learned band can move with the rhythm.

Use machine learning when the anomaly lives in a combination of signals. Fraud, cyber incidents, equipment wear, and customer behaviour changes rarely announce themselves through one neat field. The model looks at the shape of several variables together.

The best setup is often layered. Use deterministic rules for what must never happen, statistical baselines for moving patterns, and machine learning where simple rules would become a brittle list of exceptions.

When do you use anomaly detection?

1. Payments and invoicing
   Flag duplicate payments, changed bank account numbers, unusual supplier amounts, or refund patterns that deserve manual review.

2. Machines and sensors
   Detect vibration, temperature, pressure, or energy patterns that point to wear before the machine stops.

3. Data quality monitoring
   Spot a feed that stops refreshing, a column that becomes mostly empty, a row count that collapses, or a sudden jump in duplicate records.

4. Cloud and application monitoring
   Catch unusual CPU use, request volume, error rates, or cloud spend without alerting every time normal daytime traffic rises.

5. Commercial signals
   Notice a product return rate, conversion rate, or churn pattern that suddenly differs from its normal baseline.

What to watch out for with anomaly detection

False alarms
If the system is too sensitive, people stop trusting the alerts. If it is too strict, real issues pass unnoticed. Plan time after launch to tune thresholds, labels, and alert routing.

Seasonality
Black Friday, school holidays, billing cycles, and end-of-month processing can all look strange to a naive detector. Use calendar context where the pattern is predictable.

Changing normal behaviour
Your business grows, customers change habits, and upstream systems evolve. A baseline trained on old behaviour can slowly become wrong. Treat this like model drift: monitor the detector itself and refresh the definition of normal.

No owner for the alert
An alert without an owner becomes background noise. Decide who checks each alert type, what evidence they need, and what happens when the anomaly is real.

Anomaly does not mean error
An outlier can be bad data, random variation, fraud, a product issue, or a genuine opportunity. The detector should start the investigation, not replace it.