AI incident

What is an AI incident?

An AI incident is any event where an AI system, or the way people used it, caused harm or came close to it. The follow-up mail that went to four hundred customers instead of forty. The price a model published a decimal too low. The document an assistant surfaced to a colleague who had no business seeing it.

The term carries two meanings, and mixing them up causes trouble in both directions. The operational meaning is the one above: something went wrong, and you need a way to notice it, stop it, repair it and learn from it. The legal meaning is much narrower. The AI Act defines a serious incident and hangs a reporting duty on it, and almost nothing that happens in an ordinary week qualifies. The OECD's expert group added a useful third word in May 2024: an incident is where harm actually happened, a hazard is where it plausibly could have. Near misses go in the same log, because they are the cheapest lessons you will get.

What the AI Act calls a serious incident

Article 3, point 49 defines a serious incident as an incident or malfunctioning of an AI system that directly or indirectly leads to one of four things: the death of a person or serious harm to someone's health, a serious and irreversible disruption of the management or operation of critical infrastructure, the infringement of obligations under Union law meant to protect fundamental rights, or serious harm to property or the environment. Read that list once and the scale is clear. A chatbot giving a customer the wrong delivery date is not on it.

Article 73 says who reports and how fast. Providers of high-risk AI systems report to the market surveillance authority of the Member State where the incident happened, immediately after the causal link is established and in any case within 15 days of becoming aware. Two exceptions shorten that: two days for a widespread infringement or a serious and irreversible disruption of critical infrastructure, and 10 days where a person has died, counted from the moment you suspect a causal relationship rather than from proof of one. An incomplete first report followed by a full one is explicitly allowed.

Most Belgian companies are deployers, not providers, so Article 26(5) is their paragraph. If you think that using the system as instructed creates a risk, you inform the provider without delay and you suspend use. If you have identified a serious incident, you inform the provider first, then the importer or distributor, then the market surveillance authority. The provider builds and reports; you watch and raise the flag. On timing, Regulation (EU) 2026/1744, the Digital Omnibus on AI, entered into force on 27 July 2026 and pushed the high-risk obligations back to 2 December 2027 for stand-alone systems and 2 August 2028 for AI built into products already covered by EU product-safety law.

What counts as an incident in an ordinary company

None of that tells you what to write down on a Tuesday. This is the set that fills a small company's log.

• A mass mail on a bad list. A query returns more rows than anyone expected and the mail goes out correctly to the wrong people.

• A wrong number published. A generated price, stock figure or delivery estimate reaches the webshop before anyone reads it.

• A leak through an assistant. Somebody asks a question and gets an answer from a file they should not have been able to open.

• An agent that acted. A booking made, a record deleted, a supplier mailed, none of it approved by a person.

• A hallucinated answer that reached a customer. An invented warranty term, a policy that does not exist, a product you do not sell.

A workable rule: log it if a customer or a colleague saw it, if money or a record moved, if personal data ended up where it did not belong, or if you had to switch something off. Anything worth pressing the agent kill switch for is worth a line in the log.

The five steps when something goes wrong

1. Notice it. Somebody has to see it and know where to say it. Google's SRE practice advises declaring early rather than waiting for certainty, with three triggers: a second team is needed, customers can see it, or an hour of concentrated work has not solved it.

2. Contain it. Stop the system before you understand it. Switch the agent off, disable the trigger, pull the page. An hour of downtime costs less than another hour of the same output going out.

3. Work out who was affected. Which records, which customers, which period. This step decides whether you have a nuisance or a notification, and it is where an audit trail earns its keep. If nothing logged what the model was shown and what it did, you are guessing.

4. Correct and tell people. Repair the data, send the correction, call the customer who got the wrong quote. A short honest mail on the day beats a polished one next week.

5. Write it down. While it is fresh, in the register, in the same shape every time.

When personal data is involved, a second clock starts

Article 33 of the GDPR gives the controller 72 hours from becoming aware of a personal data breach to notify the supervisory authority, unless the breach is unlikely to result in a risk to people's rights and freedoms. In Belgium that goes to the Gegevensbeschermingsautoriteit through its portal, and the authority says plainly that reports sent by email will not be handled. Article 34 adds that where the risk is high you have to tell the people concerned too, and Article 33(5) requires you to document every breach, including the ones you decided not to report and why.

A small incident handled well

A forty-person installation company runs an internal assistant over its SharePoint. On a Tuesday morning a project engineer asks it about salary scales for a quote and gets back a table with the individual wages of the four site foremen, out of an HR folder.

He tells the office manager within the hour. She disables the SharePoint connector, ten minutes of work, and the assistant keeps answering from its other sources. Nobody yet knows why it happened, and that is fine. The logs then get checked for every question that touched the HR site: three in six weeks, all from the same engineer, all this morning's topic. Nothing left the company, and the four foremen whose wages appeared are told the same day.

The postmortem finds the cause, and it is not the model. The search index had been built with an administrator account, so the assistant could read sites its users could not. Three things change: the index runs under an account with the rights of a normal employee, the HR site is excluded explicitly, and a monthly report lists which sites are indexed and under whose rights. On the GDPR question the company documents the breach and reasons its way to not notifying: internal exposure, four people concerned, contained within the hour, no copies made. Had the same index been feeding a public chatbot, the answer would have gone the other way and the 72 hours would have started at nine that morning.

The register and the postmortem

The register is a table, and a spreadsheet is a fine start. One row per incident: when you noticed it, which system, what happened in one sentence, who or what was affected, what you did immediately, who owns the follow-up, and where the postmortem lives. Nothing more, because a form nobody fills in is worse than no form. It only works if it is tied to a list of what you actually run, so an agent registry or AI inventory gives every entry a system name that means something and answers the question that always comes second: where else is this thing running.

Google's SRE book defines a postmortem as a written record of an incident, its impact, the actions taken to resolve it, the root causes and the follow-up actions that stop it happening again. What makes it work is blamelessness: you assume everyone acted sensibly with the information they had, and you go looking for why that information was wrong or missing. As the same book puts it, you cannot fix people, but you can fix systems and processes.

The test of a postmortem is its action items. A permission narrowed, a limit set, an approval step added, an example added to the test set: those are fixes. "We will be more careful with the HR folder" is a reminder with a six-week shelf life. If the follow-up is something somebody has to remember, the incident comes back under a different name.

An AI incident versus an ordinary IT incident

The steps look the same, so the two get treated as one thing. They part company on what fixed means.

An ordinary IT incident has a defect at the bottom of it. The disk filled up, a certificate expired, a query joined on the wrong column. You find the line, you change it, the same input now produces the right output every time, and you can prove it with a test that fails before the change and passes after.

An AI incident often has no line to fix. The model produced a bad answer for an input it handled fine last week and may handle fine next week. Nothing broke. The behaviour is a distribution rather than a rule, so "it will not happen again" is a claim you cannot honestly make. What you can change is the setting the model works in: what it may reach, what context it gets, which version it runs, and who confirms before an action leaves the building. The fix moves outward from the model, not into it, and your regression test becomes a set of real examples, including the one that just bit you, that you rerun on every prompt change and every model upgrade.

So closing an AI incident means naming the class of output that went wrong and what now catches it, not pointing at a patched line. Containment also carries more weight, because you cannot count on a fix landing quickly. For anything that mails customers or writes to your ledger, the control you can rely on is the human confirmation in front of the action, plus a way to stop the system while you think.