When you type a question to an assistant such as ChatGPT, Claude or Gemini and get a fluent answer back, it is natural to assume that something on the other side "understands" the way a person does. In reality the basic mechanism is surprisingly simple to describe, even though it is enormous to build: a large language model (LLM) is a system that predicts which piece of text comes next. Grasping this principle helps you use it better and trust it in the right way.
From text to tokens
A model does not read letters or whole words: it works on tokens, which are fragments of text. A token can be a short word ("house"), part of a word ("intellig", "ence") or a punctuation mark. The sentence "The cat sleeps on the sofa" may become six or seven tokens, depending on how the model's vocabulary was built.
Each token is turned into a list of numbers called a vector, and the model performs its calculations on those numbers. This explains a few oddities you may have noticed:
- models sometimes miscount the letters in a word, because they do not "see" individual letters but chunks;
- paid services bill usage in tokens, not words;
- languages that are less represented in the data tend to need more tokens to express the same idea.
Predicting the next word
At the heart of it all is a question repeated endlessly: given all the preceding text, which token is most likely now? The model computes a probability distribution over its entire vocabulary. If the text is "The capital of France is", the token "Paris" will have a very high probability, while "Lyon" will have a much lower one.
The model picks a token, appends it to the text and runs the calculation again. A three-hundred-word answer is therefore the result of hundreds of consecutive predictions. There is no "ready-made answer" fetched from an archive: every sentence is assembled one piece at a time.
The role of temperature
If the model always picked the most likely token, answers would be repetitive and predictable. That is why a parameter called temperature is used: low values make choices more conservative and consistent, high values add variety. It is the reason the same question asked twice can produce different answers.
How it learns: training
To become good at prediction, the model goes through distinct phases.
- Pre-training. The model reads a huge amount of text and, for each fragment, tries to guess the next token. When it is wrong, its internal parameters (billions of numbers called weights) are nudged slightly. Repeating this an immense number of times, the model absorbs grammar, style, recurring facts and patterns of reasoning found in the texts.
- Instruction tuning. A model that can only continue text is not yet a good assistant. It is therefore trained on examples of well-made questions and answers, so it learns to follow requests, reply in an orderly way and decline inappropriate tasks.
- Learning from human judgement. In many pipelines, people compare pairs of answers and pick the better one; those preferences are then used to steer the model's behaviour further.
Context: the working memory
Everything the model "knows" during a conversation lives in the context window: the chat text, the system instructions, any attached documents. The model does not remember past conversations on its own; if a service seems to remember them, it is because the application feeds that information back into the context.
Inside the context, the attention mechanism comes into play, introduced with the architecture known as the Transformer. Put simply, for each token the model weighs how relevant every other token in the text is. In the sentence "I put the book in the backpack because it was light", attention helps link "it was light" to "book" or "backpack" based on the overall context.
What a language model is not
Understanding the mechanism also helps you avoid wrong expectations:
- it is not a search engine: unless it is connected to external sources, it answers with what it absorbed during training, which may be incomplete or out of date;
- it is not a database of verified facts: it produces plausible text, which usually matches the truth but not always;
- it has no intentions of its own: it follows learned patterns and the instructions it receives.
| Feature | Language model | Search engine |
|---|---|---|
| What it returns | Newly generated text | Links to existing pages |
| Source | Learned parameters | Index of documents |
| Updates | Up to the training date | Continuous |
| Typical risk | Made-up statements | Low-quality sources |
Why it works so well
It may seem strange that "predicting the next word" is enough to write code, summarise contracts or explain physics. The point is that, to predict human text well, the model has to build useful internal representations of concepts, relationships and procedures. More data, more parameters and better training lead to richer representations. It is not understanding in the human sense, but it is a very powerful form of statistical competence.
In short
- A language model generates text by predicting one token at a time based on context.
- It learns from vast amounts of text and is then refined to follow instructions.
- The context window is its only memory during a conversation.
- It produces plausible text, not necessarily true text: important information must be checked.
