The synthetic voices of old satellite navigators were instantly recognisable: they pronounced words mechanically, with unnatural intonation. Today a speech synthesis system can read an article with believable pauses, emphasis and breaths, and other models compose music from a text description. To understand how we got here, you need to start from how a computer represents sound.
Sound as numbers
Sound is a vibration of the air. To digitise it, a microphone measures its amplitude many thousands of times per second: CD-quality audio uses 44,100 samples per second, and speech for synthesis applications is often handled at lower sampling rates. A single second of audio is therefore a sequence of tens of thousands of numbers, much longer than a written sentence.
That length is the main difficulty: generating every sample directly, one after another, is possible but expensive. This is why modern systems often work on more compact representations.
The spectrogram
A spectrogram is a picture of sound: the horizontal axis shows time, the vertical axis frequency, and the intensity of each point indicates how much energy there is at that frequency at that moment. Vowels appear as horizontal bands, consonants as short bursts of noise. Many systems use a variant called the mel spectrogram, which redistributes frequencies in a way similar to how the human ear perceives them.
How speech synthesis works
A speech synthesis (text-to-speech, TTS) system usually follows this sequence.
- Text normalisation. Numbers, abbreviations and symbols are converted into words: "3 kg" becomes "three kilograms", "Dr." becomes "doctor". This step is less trivial than it seems, because "1/2" could be "one half" or "the first of February".
- Conversion to phonemes. The text is translated into the sounds of the language. Some languages have fairly regular spelling-to-sound rules, but there are always tricky cases, such as words spelled the same but stressed differently.
- Acoustic model. A neural network generates, from the phonemes, a spectrogram describing how the sentence should sound, including rhythm and intonation, known as prosody.
- Vocoder. A second model, called a neural vocoder, turns the spectrogram into an audible sound wave.
Many recent systems merge these steps into a single model trained end to end, but the underlying logic remains the same.
The leap in quality: audio tokens
An important innovation is neural audio codecs: models that compress sound into a sequence of discrete symbols, similar to text tokens, and can then reconstruct it with good fidelity. Once audio has been turned into tokens, architectures similar to those of language models can be used: instead of predicting the next word, the model predicts the next fragment of sound. This approach has made synthetic voices far more natural and has also made it possible to generate non-verbal sounds, such as laughter or hesitation.
Music generation
Music poses further challenges compared with speech:
- long structure: a piece has verses, choruses and variations that must stay consistent for minutes;
- several simultaneous voices: different instruments play together and must harmonise;
- rhythm: small timing errors are noticed immediately.
There are two main approaches. The first generates symbolic music, that is notes and durations, like a digital score, which is then played by virtual instruments. The second generates audio directly, using audio tokens or diffusion models applied to spectrograms. The second produces richer results but is harder to control in detail.
| Approach | What it generates | Advantage | Limit |
|---|---|---|---|
| Symbolic | Notes and durations | Easy to edit and arrange | Sound depends on virtual instruments |
| Direct audio | Waveform or spectrogram | Realistic timbres and mix | Hard to correct a single note |
Legitimate and useful applications
- Accessibility: reading texts aloud for people with visual impairments or reading difficulties.
- Audiobooks and training content produced in several languages.
- Voice assistants and phone systems that are easier to understand.
- Musical prototyping: trying out an arrangement idea before recording it.
- Voice preservation for people at risk of losing their voice through illness, starting from earlier recordings.
Consent over one's voice
A voice is a personal trait. Reproducing a real person's voice requires their explicit consent, and reputable services check for it. In addition, the AI Act lays down transparency obligations for artificially generated audio content: those who publish it must make its synthetic nature identifiable in the ways prescribed. These are common-sense rules that protect both people and trust in content.
In short
- Digital audio is a very long sequence of samples, often represented as a spectrogram.
- Speech synthesis goes through text normalisation, phonemes, an acoustic model and a vocoder.
- Neural audio codecs turn sound into tokens and have made voices much more natural.
- Music is generated either symbolically or directly as audio, with different advantages.
- Using a real person's voice requires their consent and transparency towards the audience.
