In the world of artificial intelligence there is an old saying among computer scientists: garbage in, garbage out. A model, however sophisticated, mirrors the data it learned from. The most widely used algorithms are public and well documented; what often separates a reliable system from a mediocre one is the barely visible, very laborious work done on the data.
What training data is
Training data is the set of examples a model observes in order to learn. Its form depends on the task:
- for an image classifier, thousands of photos with a note of what they contain;
- for a system that estimates delivery times, the order history with times, distances and delays;
- for a language model, huge collections of text, followed by examples of well-crafted conversations;
- for a transcription system, audio recordings paired with the matching text.
Where it comes from
There are four common sources, each with its own advantages and problems.
- Internal data. Management software, CRM records, support tickets, machine logs. Relevant, but often messy and incomplete.
- Public datasets. Collections released by universities, institutions and research communities. Handy for getting started, but not always representative of your case.
- Data collected from the web. Used mainly for large language models. Plentiful, but of very variable quality, with open questions about copyright and personal data.
- Synthetic data. Artificially generated examples, sometimes produced by other models. Useful for covering rare cases, but best used in moderation: a model trained mostly on synthetic data risks amplifying its flaws.
The most frequent dataset flaws
Wrong or inconsistent labels
If two people label the same complaint as "billing" and "shipping" respectively, the model receives contradictory signals. Even modest proportions of wrong labels can noticeably worsen performance, especially in the smaller categories.
Class imbalance
In an archive of bank transactions, fraud cases are very few compared with regular payments. A lazy model could always answer "regular" and achieve an apparently very high accuracy without catching a single fraud. That is why, besides accuracy, metrics such as precision and recall are measured.
Unrepresentative data
A speech recognition system trained almost entirely on voices from one region will understand other accents less well. A model for predicting failures, trained on new machinery, might not recognise the typical warning signs of older equipment.
Information leakage
Data leakage occurs when the training data contains information that, in real use, would not be available at prediction time. A classic example: to predict whether a customer will cancel a subscription, the field "contract closing date" is included by mistake. The model looks infallible during testing and fails as soon as it goes into production.
Stale data
Habits change. A sales forecasting model trained on years very different from the current one can give misleading guidance. This phenomenon, called data drift, means models must be monitored and retrained periodically.
How quality data is prepared
Preparation usually follows a few recurring stages:
- cleaning: removing duplicates, fixing formats (dates, currencies, units of measure), handling missing values explicitly;
- labelling guidelines: a clear document, with examples and edge cases, that labellers must follow;
- cross-checking: having part of the examples labelled by several people and measuring how much they agree;
- balancing: collecting more examples of rare cases or weighting them differently during training;
- documentation: recording where the data comes from, when it was collected, what it contains and what is missing.
| Problem | Typical symptom | Countermeasure |
|---|---|---|
| Inconsistent labels | Unstable performance | Guidelines and double review |
| Imbalanced classes | Rare cases never recognised | Suitable metrics, rebalancing |
| Data leakage | Great in testing, poor in use | Check what is known at prediction time |
| Stale data | Gradual drop in accuracy | Monitoring and retraining |
Personal data and rights
When data concerns people, specific rules apply. In the European Union the GDPR requires a legal basis for processing, respect for the principle of data minimisation and transparency towards the people concerned. In practice it pays to always ask whether a field is really needed: name, address or phone number often add nothing to model quality and only increase risk. Pseudonymising data before training is also a good habit.
More data is not always better
Adding examples helps as long as they are relevant and correct. A dataset of ten thousand clean, well-labelled examples can deliver better results than one ten times larger but full of noise. Before collecting more, it is almost always worth improving what you already have.
In short
- A model mirrors the data it learns from: errors in the data become errors in the predictions.
- The most common problems are inconsistent labels, imbalanced classes, unrepresentative data, information leakage and stale data.
- Cleaning, labelling guidelines, cross-review and documentation are the main defences.
- With personal data you need a legal basis and minimisation, as the GDPR requires.
