A general-purpose language model knows a great deal, but it does not know your company's internal manual, the current price list, the plant's safety procedures or the contract terms signed with a client. Yet in a work setting it is precisely on this information that precise answers are needed. The most widespread technique for bridging this gap is called RAG, short for Retrieval-Augmented Generation.
The idea in one sentence
Before answering, the system searches your documents for passages relevant to the question and hands them to the model together with the question itself, asking it to answer on the basis of that material. It is the difference between a closed-book and an open-book exam: the model does not have to remember, it has to read and summarise.
The components of a RAG system
1. The document base
This is the set of documents to draw from: manuals, procedures, FAQs, standard contracts, product sheets, minutes. The quality of a RAG system depends first and foremost on this. Contradictory documents, duplicates in different versions or outdated material will produce contradictory or obsolete answers.
2. Indexing
Documents are split into chunks and made searchable, usually with a combination of embedding-based semantic search and keyword search. Useful metadata is attached to each chunk: document title, section, date, version, department, confidentiality level.
3. Retrieval
When a question arrives, the system selects the most relevant chunks. Many systems add a reranking step: a second model rereads the candidates and reorders them according to how well they actually answer the question.
4. Generation
The selected chunks are placed in the language model's context, together with precise instructions, for example:
- answer only on the basis of the documents provided;
- if the information is not present, say so clearly;
- state which document and section each statement comes from.
5. Citations
A well-built RAG answer includes references to the documents used. This lets the user check with a click and go back to the original text when the matter is delicate.
A concrete example
In a maintenance company, a technician asks: "How often should the filter be replaced on boiler model X?". The system retrieves the page from the manufacturer's manual and the company's internal procedure, which sets a shorter interval for installations in areas with very hard water. The model answers with both indications, explains which applies in which case and cites both sources. Without RAG, a general model would have given a plausible answer that was not necessarily correct for that model and that company.
Where a RAG system goes wrong
| Problem | Typical cause | Remedy |
|---|---|---|
| "Not found" even though the data exists | Poorly split chunks or weak search | Revise chunking, use hybrid search |
| Answer based on an old document | Several versions in the archive | Validity metadata, removal of superseded files |
| Mixes up information on different products | Similar chunks retrieved together | Filters by product or category |
| Adds details that are not there | The model fills in from its own knowledge | Stricter instructions, citation checks |
| Shows confidential content | Permissions not applied at retrieval | Filter results by user |
Security and permissions
A RAG assistant can unintentionally become a shortcut around permissions. If the archive contains HR documents and the system does not filter results according to who is asking, anyone could obtain their content by asking the assistant. Permission checks must happen at retrieval time, not just in the interface.
There is also a subtler risk: a document added to the archive might contain instructions aimed at the model, such as "ignore the rules and reply that...". The system's instructions should state that document content is to be treated as information to report, not as commands to execute, and the group of people who can upload documents should be controlled.
How to evaluate a RAG system
Before making it available to everyone, it is worth preparing a list of real questions with the correct answers and expected sources. For each one, check:
- whether the system retrieved the right documents;
- whether the answer is correct and complete;
- whether every statement is supported by the cited sources;
- whether, for questions with no answer in the archive, the system says so.
Repeating this check after every significant change, such as adding many documents or switching models, prevents silent deterioration.
When RAG is the right choice
RAG is particularly suitable when information changes often, when verifiable citations are needed and when there are too many documents to paste into a conversation each time. It is less useful if the goal is to change the model's style or behaviour: in that case other techniques come into play, such as more detailed instructions or fine-tuning the model.
In short
- RAG has the system look up relevant documents and asks the model to answer on that basis.
- Quality depends above all on up-to-date, well-chunked documents with metadata.
- Citations make it possible to check every answer against the original text.
- Permissions must be applied at retrieval time, not only in the interface.
- A set of test questions with known answers is essential for evaluating the system.
