The market for artificial intelligence services is crowded: general-purpose assistants, vertical tools for lawyers, accountants or agencies, platforms promising to automate customer support. Demonstrations are almost always convincing, because they show the cases where the product works best. To decide sensibly you need a method that puts the service to the test on your problems, with your data and under your conditions.
1. Define what it must do, concretely
Before looking at any product, write down in a few lines the task it must perform and how you will tell whether it performs it well. For example: "classify incoming emails into the office's six categories, getting fewer than one in twenty wrong, and flag ambiguous ones as uncertain". Without a measurable goal, evaluation becomes a matter of impressions.
2. Test on real cases, not on the demo
Prepare a test set of a few dozen examples taken from your daily work, including:
- simple, frequent cases;
- difficult or ambiguous cases;
- edge cases that cause problems even for people today;
- examples where the correct answer is "I don't know" or "a human is needed".
You already know the right answer for each, so you can compare results objectively. Use anonymised data if the service has not yet been approved for real data. Repeat the test several times: generative models can give different answers to the same question.
What to look for
- Accuracy on the answers you know.
- Handling of uncertainty: does the service admit when it does not know, or does it make things up?
- Consistency across repeated tests.
- Response times under realistic conditions.
- Ease of correction: how much work does it take to fix a wrong result?
3. Examine how data is handled
Here you need written answers, not verbal reassurances.
| Question | Answer you want to find |
|---|---|
| Is my data used to train the models? | No, or only with explicit, revocable consent |
| Where is it processed and stored? | Clearly stated locations, with adequate safeguards |
| How long is it kept? | A defined period and the option to delete |
| Who can access it at the provider? | Limited, logged access |
| Is there a data processing agreement? | Yes, compliant with the GDPR |
| Are there sub-processors? | A list is available and kept up to date |
If the service will process personal data on behalf of your organisation, the provider acts as a data processor and a contract governing this is required.
4. Assess security
- Does the service support multi-factor authentication and, for businesses, sign-in through the company identity system?
- Can permissions be managed per user or group?
- Are activity logs available for review?
- If the service connects to email, documents or business software, what authorisations does it require? Are they proportionate?
- Does the provider explain how it protects the system against misuse, such as hidden instructions in the documents it processes?
5. Work out the real cost
The list price is only part of the cost. Consider:
- the pricing model: per user, per usage, per volume of documents;
- variable costs that can grow quickly with use;
- set-up and training time;
- time spent reviewing results;
- any integrations with existing systems.
Then compare this total with the time actually saved during the test, not with the time promised.
6. Read the clauses that matter
In the contract or terms of use, look in particular for:
- who owns the rights to generated content;
- limitations of the provider's liability in case of errors;
- guaranteed service levels, such as availability and support response times;
- notice of substantial changes to the service or the underlying model;
- termination conditions.
7. Plan your exit
This is the question most often forgotten: what happens if, a year from now, you want to change provider? Check that you can export data, configurations, histories and documents in standard formats. A service that makes leaving difficult exposes you to dependence, price rises and disruption.
8. Consider regulatory obligations
If the service will be used in sensitive areas, such as recruitment or creditworthiness assessment, the AI Act may classify it as a high-risk system, with obligations for those using it as well. Ask the provider where it stands in relation to the regulation and what documentation it makes available.
A simple scoring method
To compare two or three services, give each area a score from 1 to 5 and a weight according to your priorities. A service that excels on quality but falls short on data handling should not win just because it has the highest total score: some requirements, such as GDPR compliance, are minimum conditions, not items to be offset.
In short
- Define a measurable goal before looking at products.
- Evaluate the service on a set of real cases whose answers you know.
- Insist on written answers about the use, location and retention of data.
- Calculate the real cost, including set-up, training and review.
- Check in advance how to export your data and leave the service.
