Behind almost every artificial intelligence system making headlines today, from machine translators to models that generate images, there is an artificial neural network. The name evokes the brain, but the comparison should be taken with caution: a neural network is a mathematical model only loosely inspired by biological neurons. Understanding how it is built removes much of the magic surrounding it and helps you judge its strengths and limits.
The artificial neuron
The basic unit is the artificial neuron, which performs three very simple operations.
- It receives numbers as input. For example, the brightness of a few pixels in an image.
- It multiplies them by weights and adds them up. Each input has a weight indicating how much it counts. A value called the bias is added to the sum, shifting the activation threshold.
- It applies an activation function. The result of the sum passes through a function that decides how strongly the neuron "fires". One of the most widely used, called ReLU, lets positive values through and sets negative ones to zero.
On its own, a neuron is little more than a calculator. The power comes from connecting thousands or billions of them.
A concrete example
Imagine a neuron that has to decide whether to take an umbrella. The inputs are: chance of rain, distance to walk, availability of a car. The weight for rain will be high and positive, the weight for the car negative (if you have a car, you need the umbrella less). The weighted sum, passed through the activation function, gives an answer. In a real network nobody sets these weights by hand: they are learned from data.
Layers
Neurons are organised in layers:
- the input layer receives the raw data;
- the hidden layers process the information in successive steps;
- the output layer produces the result, for example the probability that an image contains a cat.
When there are many hidden layers we speak of deep learning. "Deep" refers precisely to the number of layers, not to any particular depth of thought.
The advantage of depth is that each layer can build more abstract representations from those of the previous layer. In a network analysing images, the first layers tend to respond to edges and contrasts, the middle ones to shapes and textures, the last ones to whole objects such as eyes, wheels or windows.
How a network learns: backpropagation
At first the weights are random numbers and the network produces nonsense. Learning happens by repeating a cycle:
- the network is shown an example, such as a photo of a dog labelled "dog";
- the network computes its prediction, perhaps "cat, 70%";
- a loss function measures how far the prediction is from the correct answer;
- backpropagation works backwards from the output layer to compute how much each weight contributed to the error;
- each weight is adjusted slightly in the direction that reduces the error, using a method called gradient descent.
Repeated over millions of examples, this cycle leads the network to configure its weights so that its answers become more and more accurate. A helpful analogy is a blindfolded person trying to walk down into a valley: they cannot see the landscape, but they can feel which way the ground slopes under their feet and take a small step in that direction, then another.
Different architectures for different tasks
Not all networks are alike. Over time, specialised architectures have emerged:
| Architecture | Strength | Typical use |
|---|---|---|
| Fully connected networks | Simplicity | Tabular data, small problems |
| Convolutional networks (CNN) | Recognising local patterns | Images, computer vision |
| Recurrent networks (RNN) | Processing sequences | Time series, text (in the past) |
| Transformer | Long-range relationships | Language models, translation |
Transformers, based on the attention mechanism, are now the dominant architecture for language and are increasingly used for images and audio too.
Why so much data and computing power are needed
A network with billions of weights needs a great many examples to set them sensibly; with little data it tends to memorise them instead of generalising. In addition, each training step requires an enormous number of multiplications. That is why GPUs are used: processors designed for graphics that run thousands of simple operations in parallel, exactly what a neural network needs.
Limits worth knowing
- Poor interpretability. It is hard to explain why a network gave a particular answer: the "rule" is spread across millions of weights.
- Sensitivity to data. If the training examples have gaps or distortions, the network will reproduce them.
- Fragility outside familiar conditions. A network trained on daytime photos may behave badly with night-time ones.
- Energy cost. Training very large networks requires expensive infrastructure.
In short
- A neural network is made of artificial neurons that sum weighted inputs and apply an activation function.
- Neurons are organised in layers; with many layers we speak of deep learning.
- The network learns by adjusting weights through backpropagation and gradient descent.
- Different architectures, such as CNNs and Transformers, suit different tasks.
- Interpretability and dependence on data remain the main limitations.
