People tend to think an algorithm is neutral by definition: after all it is mathematics, and mathematics has no preferences. Yet artificial intelligence systems can produce results that are systematically unfavourable to certain groups of people. This phenomenon is called algorithmic bias. It almost never stems from an explicit intention, but from choices and data that carry real-world distortions with them.
What bias is
In statistics, bias is a systematic error: a tendency to be wrong always in the same direction. In the context of AI the term refers mainly to cases in which a system treats people differently, without valid justification, because they differ in characteristics such as gender, age, origin, disability or area of residence.
The key point is that a model learns to reproduce the patterns present in the data. If those patterns reflect past inequalities, the model learns them and applies them at scale.
Where it comes from
Distorted historical data
Imagine a system that helps screen candidates, trained on a company's hiring decisions over recent years. If in the past certain roles were filled almost exclusively by people of a particular profile, the model will learn that this profile is "the ideal candidate", even if the reason was prejudice rather than competence.
Unrepresentative data
A speech recognition system trained mainly on adult voices will understand children and older people less well. A model for analysing images of skin, trained on a limited range of skin tones, may be less accurate on others.
Proxy variables
Even if a sensitive characteristic is excluded, the model may reconstruct it from other correlated information. A postcode, for example, can indirectly reflect origin or economic status. Removing the "nationality" column is not enough to guarantee fairness.
Design choices
What you decide to predict matters. If past spending is used to estimate the need for care, you risk underestimating those who had less access to services rather than those who needed them less.
The feedback loop
The model's decisions influence future data. If a system concentrates checks on a certain area, more irregularities will surface there, reinforcing the belief that the area is more problematic.
Bias in language models
Models that generate text can also reflect stereotypes present in the texts they learned from. A simple example: asked to write a story about "a doctor and a nurse", the model might assign genders in a stereotyped way. Providers apply techniques to mitigate these effects, but it is worth being aware of them when using these tools for texts aimed at the public.
How it is measured
There is no single definition of fairness, and some definitions conflict with each other. The most commonly used compare the model's behaviour across different groups.
| Criterion | Question it asks |
|---|---|
| Outcome parity | Do different groups receive positive outcomes in similar proportions? |
| Error parity | Does the model make mistakes equally often across groups? |
| Calibration | Does a risk score mean the same thing for all groups? |
Choosing which criterion to prioritise is not just a technical decision: it depends on the context and on the consequences of errors, and it should be discussed with domain experts and with the people affected.
How to reduce it in practice
- Examine the data before training. Check how the different groups are distributed and whether some are absent or under-represented.
- Evaluate the model by subgroup. An overall accuracy of 95% can hide 80% on a specific group.
- Rethink the objective. Ask whether the variable being predicted really measures what you care about.
- Involve diverse people in design and testing: those with different experiences notice problems others miss.
- Keep human oversight in decisions affecting people, with the possibility of challenging the outcome.
- Monitor after release. Bias may only emerge in real use.
- Document choices, known limitations and the results of checks.
The regulatory framework
In the European Union the issue is addressed by several laws. The GDPR, in Article 22, recognises the right not to be subject to decisions based solely on automated processing that produce legal effects or similarly significantly affect a person, subject to exceptions and safeguards. The AI Act, for high-risk systems, imposes requirements on data quality and human oversight precisely to limit these effects.
In short
- Algorithmic bias is a systematic error that penalises some groups without justification.
- It stems from distorted historical data, unrepresentative data, proxy variables and design choices.
- There are several fairness criteria, to be chosen according to context.
- Subgroup analysis, human oversight and continuous monitoring are the main defences.
- The GDPR and the AI Act provide specific safeguards for automated decisions.
