Bias

What is bias?

Bias in AI means that a model is prejudiced. It treats certain groups or situations differently, usually because the data it was trained on already carries those prejudices. As a result, an AI can quietly reinforce inequality instead of making objective decisions.

How does bias arise?

Bias can creep in through several routes:

• Data bias: the training data is not representative, or it carries old patterns of inequality.

• Model bias: the algorithm learns the wrong relationships or generalises too aggressively.

• Human bias: choices made by developers, such as which data to use or how results are interpreted.

Example

Amazon once built an AI system to screen CVs. The model gave men consistently higher scores than women, because it had been trained on historical hiring data in which men were hired far more often. The model learned the pattern and repeated it.

Is bias always bad?

Not always. Some bias is unavoidable, because every model learns from data. It only becomes a problem when the skew leads to unfair or discriminatory outcomes. A model can be deliberately tuned for children, for example, if it is meant to detect childhood diseases. That is bias on purpose, and it is appropriate.

How do you detect and limit bias?

1. Analyse your data. Check whether all relevant groups are well represented. Look beyond raw counts and inspect data quality too. Some groups may be underrepresented, or their data may contain more errors. Both can pull a model off course.

2. Test by group. Measure whether the model performs equally well for everyone. Compare how often it gets predictions right across ages, regions, or genders. Large gaps are a strong signal of bias.

3. Use fairness metrics. These are specific measures for checking whether a model produces fair outcomes. You compare how often the model makes correct decisions for different groups, or whether everyone has the same chance of a positive outcome. Two well-known examples are demographic parity (equal positive rates per group) and equal opportunity (equal chance of a correct positive prediction). They make uneven treatment visible quickly.

4. Document your choices. Write down why you picked certain data or models, what assumptions you made, and what you changed after testing. That paper trail makes it much easier to revisit decisions later or justify them to an auditor.

5. Keep humans in control. Never hand sensitive decisions over to AI completely. Use AI as a tool and make sure a person can take the final call, especially when the decision has personal or social impact.

The European legal context

The European AI Act applies to any organisation that puts AI into use within the EU, regardless of where the company is based. It requires active monitoring of bias, especially in high-risk applications such as recruitment, education, healthcare, or credit scoring.

The GDPR (General Data Protection Regulation) sits alongside it. GDPR requires organisations to handle personal data carefully, only use data that is genuinely relevant, and give people the right to an explanation of automated decisions. It also says people should not be judged purely on the basis of an algorithm when the decision has significant impact on them, for example a credit application or a job offer.