Jagged intelligence

What is jagged intelligence?

Jagged intelligence is the pattern where an AI model handles a hard task well and then fails at an easy task right next to it. The same assistant that drafts a solid supplier contract can get a VAT total wrong. Which side of the line a task lands on has little to do with how hard a person would find it.

That last part is the whole point. With a human colleague you can extrapolate: someone who can do the difficult version of a job can do the simple version. With a language model that rule does not hold. Capability comes in peaks and holes, and the holes sit in places you would not think to check.

The picture Ethan Mollick uses for it is a fortress wall with towers of uneven height. Inside the wall the model is reliable, outside it is not, and the wall zigzags through the middle of tasks that look identical to you.

How jagged intelligence arises

Two things decide where the peaks are, and neither of them is human difficulty.

How much of the task is in the training data. A model gets good at what it has seen in bulk. Contracts, code, marketing copy: the internet is full of them. A specific calculation under a Belgian VAT rule shows up far less often, so the model's grip is weaker, even though a bookkeeper would call it routine.

Whether the task can be checked automatically. In recent years the big labs have trained their models heavily with reinforcement learning: the model tries a task, an automatic checker scores the answer, and the model learns from the score. That only works where a checker exists. Maths problems and code have an exact answer, so the model can practise millions of them. Judging whether an email sounds right to a client, or whether it is quicker to walk or drive fifty metres, has no checker, so the model gets no such practice. Andrej Karpathy, the AI researcher who coined the term, put it this way in 2026: older software automates what you can specify in code, current models automate what you can verify.

Our intuition about difficulty is calibrated on people, where skills grow together: nobody learns calculus before they can count. A model's skills were never forced to grow together.

Where the term comes from

In 2023 a team from Harvard Business School, Wharton, MIT Sloan and Warwick, with Ethan Mollick among the authors, ran an experiment with 758 staff at Boston Consulting Group. On 18 realistic tasks from their daily work that sat inside the model's capability, the people using GPT-4 finished about 12 percent more tasks, worked about 25 percent faster and delivered work rated more than 40 percent higher in quality. On one business problem chosen to sit outside that capability, the group with AI was 19 percentage points less likely to reach the right answer than the group without. The authors called this uneven boundary the jagged technological frontier.

In July 2024 Andrej Karpathy gave the same phenomenon the shorter name jagged intelligence, after a top model solved advanced maths and still argued that 9.11 is larger than 9.9.

Stanford's 2026 AI Index brought it to a wider audience. Models from the same generation that reached gold-medal level at the 2025 International Mathematical Olympiad read a clock with hands correctly about half the time, where people manage it nine times out of ten.

Two examples you will recognise

The contract and the VAT line. Ask an assistant to draft a service agreement with a Belgian supplier and you get something a lawyer would recognise as a serious first version: clauses on liability, termination, confidentiality, all in the right order. Then give it a restaurant bill of 345 euro including VAT and ask how much VAT is in it. A Belgian restaurant charges 12 percent on the meal and 21 percent on the wine. If the meal was 200 euro and the wine 100 euro before VAT, the VAT is 24 plus 21, so 45 euro. A model that was not told the split can simply divide by 1.21 and answer 59.88 euro, confidently and wrong. The contract was the hard task. The bill was the easy one.

The chart and the clock. Show a model a screenshot of a dashboard and ask what happened to margin in the second quarter. It reads the bars, picks out the dip and links it to the note in the corner. Show it a photo of a wall clock and ask the time, and it can be hours off. Charts are everywhere in the training data. Hands on a dial are rare in it, and nothing checked the answer, so the model never practised.

Jagged intelligence versus hallucination

A hallucination is one wrong output: an invented citation, a made-up figure, a policy that does not exist. Jagged intelligence is a property of the model's capability map: it tells you in which zones wrong outputs are likely at all. Hallucination is the symptom you catch in a single answer. Jaggedness is the reason the symptom clusters around certain tasks and stays away from others.

The practical difference is in what you do about it. Against hallucination you work per answer: grounding, source documents, a person who checks. Against jaggedness you work per task: you find out where the holes are before you rely on the model there, and you decide per task how much checking it needs.

The two are also connected through verification. The tasks where a model is strongest are largely the tasks with a built-in check, so those are the ones where a wrong answer gets caught fastest. The holes sit in tasks nobody checks automatically, which is exactly where a confident wrong answer travels furthest.

What it means when you roll out AI in an SME

1. Pilot per task, not per department. "AI for finance" is not something you can test. "Draft the first reminder for an overdue invoice" is. So is "assign incoming supplier invoices to a cost centre". One will be inside the frontier, the other may not be, and only a separate trial for each tells you which.

2. Write a small eval per task. Collect twenty to fifty real cases with a known correct answer, run them through the model, and count. This is the same discipline described in the evals entry, applied to one task. A task that scores 95 percent on your own cases can be automated with a light review. One that stays at 70 percent stays with a person.

3. Do not extrapolate from a demo. A vendor demo shows you the peak. That the model impressed you on a difficult example says nothing about the simple one next to it. Bring your own ugly cases to the demo: the invoice with two VAT rates, the email in dialect, the spreadsheet with merged cells.

4. Put the verification where the holes are. A task with an automatic check (a total that must add up, code that must run) can go faster with less review. A task without one (a judgement call, a tone, a legal interpretation) gets a person in the loop, however fluent the output looks.

5. Re-run the evals after each model update. The frontier moves with every release, usually outward, sometimes sideways. A task that sat in a hole in spring can be fine in autumn, and once in a while the reverse. Keep the test set and date the results.

What to watch out for with jagged intelligence

The most expensive mistake is carrying trust from one task to the next. The model nailed the difficult analysis on Monday, so on Tuesday nobody rechecks the simple total. The experiment at Boston Consulting Group showed that mechanism in numbers: on the task outside the frontier, the people with AI did worse than the people without. The model's answer was wrong and read as right.

The second is fluency. Output from a hole looks exactly like output from a peak. Same tone, same structure, same confidence. There is no signal in the text itself. The only signal is your own eval, run beforehand.