Explainable AI (XAI)

What is explainable AI?

Explainable AI, often shortened to XAI, is a set of methods that make a model's decisions understandable to a person. Rather than accepting a score with no reasoning behind it, you get to see which inputs pushed the result one way or the other.

Many machine learning models are hard to read from the inside. A model can weigh thousands of inputs in ways no human can follow line by line, which is why they are often called black boxes. XAI does not open the box fully. It gives you reliable ways to ask "why this answer?" and get a useful reply.

Take a loan application that a model rejects. On its own, the model returns a number. An explanation method can add that the rejection was driven mostly by a short credit history and a high level of existing debt, and that the applicant's income pushed slightly the other way. Now the decision can be checked and explained to the customer.

Why explainable AI matters

When a model affects someone's money, health, or job, "the computer said so" is not good enough. People need to know a decision was made for sound reasons, and they have a growing legal right to an explanation.

Explanations also help the team that builds the model. If a medical model turns out to be reading the hospital's name off a scan rather than the actual tissue, that is a serious flaw hiding behind good-looking accuracy. XAI is often how such shortcuts get caught. It is also a practical tool for spotting bias, by showing when a protected characteristic is quietly steering the results.

How explainable AI works

Most methods answer one of two questions. A global explanation asks which inputs matter most to the model overall. A local explanation asks why the model decided one specific case the way it did. Both are useful, and they can disagree.

The most common building block is feature importance: a ranking of which inputs moved the prediction, and by how much. Two methods dominate here.

SHAP (Shapley additive explanations) borrows an idea from game theory. It treats each input as a player on a team and works out how much each one contributed to the final result, in a way that adds up cleanly to the prediction. It is thorough and consistent, but can be slow on large models.

LIME (local interpretable model-agnostic explanations) takes a different route. To explain one prediction, it changes the input slightly many times, watches how the output shifts, and fits a simple, readable model to that small neighbourhood. It is faster and works on almost any model, but its explanations can vary between runs.

Both are model-agnostic: they treat the model as a box they probe from the outside, so they work on almost anything, from a credit scorecard to a deep neural network. Neither replaces basic checks like a confusion matrix, which shows where a classifier's right and wrong answers land. XAI tells you why a decision was made; a confusion matrix tells you how often the model is right.

Explainable AI and regulation

Explainability is moving from good practice to legal requirement. Under the EU AI Act, systems judged high risk, such as those used in hiring or credit scoring, must be transparent enough that their output can be interpreted and overseen by a person. The GDPR already gives people a right to meaningful information about automated decisions that significantly affect them.

This is why explainable AI is a core part of any responsible AI programme. You cannot be accountable for a decision you cannot explain.

What to watch out for with explainable AI

An explanation is not the real reason. Methods like SHAP and LIME give a plausible account of a decision, not a readout of the model's actual internal logic. Two methods can point at different inputs for the same case. Treat an explanation as a strong hint, not a confession.

Simple explanations can mislead. A neat bar chart of "top factors" feels convincing, which is exactly the risk. A tidy explanation of a shaky model can give false confidence to everyone who sees it.

Explaining is not fixing. Finding that a model leans on a biased input tells you there is a problem. It does not solve it. The explanation is the start of the work, not the end.