AI engineering / Advanced / 03.30
Giving your app a voice
How computer voices are made, what you can choose, how to write text that sounds right out loud, and the rules for using AI voices fairly.
13 min readFree
Giving your app a voice
The last lesson, Speech-to-Text, ended with a promise: next, text-to-speech, giving your app a voice. Here it is. Speech-to-text let your app listen. Text-to-speech lets it talk back.
By the end you will know, in plain words, how a modern computer voice is made, which choices you get when you ask for one, how to write text that sounds good out loud, and two rules every builder should follow. You will also run two small programs.
Key words
- Text-to-speech (TTS)
- A program that turns written text into spoken audio. Also called speech synthesis.
- Voice
- The way the speech sounds: its pitch, tone and accent. A service usually offers a list of ready-made voices.
- Audio format
- The kind of file the sound is saved in, such as MP3 or WAV.
- Streaming
- Sending the audio in small chunks, so playback can start before the whole thing is ready.
- Latency
- The wait between asking for something and getting the first piece back.
- SSML
- Speech Synthesis Markup Language. A standard way to add notes to your text, such as where to pause.
- Sample
- One tiny measurement of a sound wave. Audio is a long list of samples.
How a computer voice is made
Think of a TTS system as two workers on a line. The first reads the page and gets it ready. The second turns that prepared text into sound.
The Wikipedia page on speech synthesis describes these as a front-end and a back-end. It says the front-end first converts "raw text containing symbols like numbers and abbreviations into the equivalent of written-out words", a step often called text normalization. It then "assigns phonetic transcriptions to each word" and marks where phrases and sentences break. The back-end, "often referred to as the synthesizer", then turns that plan into sound.
Older systems stitched together small recordings of real speech. Newer ones learn from examples. The same page says deep learning speech synthesis "uses deep neural networks (DNN) to produce artificial speech from text", and that these networks are "trained using a large amount of recorded speech" together with the matching text. It dates the start of this field to September 2016, when DeepMind released WaveNet. A 2018 system, Tacotron 2, converted text into mel-spectrograms (a picture of how sound energy is spread over time) and then used a separate neural vocoder to turn that into a waveform.
You do not need to build any of that. As an AI engineer you ask a service to do it and you choose a few settings. The rest of this lesson is about those choices and about getting your text ready, which is the part you control.
The choices you get
Every TTS service is a little different, so read its docs. Still, the same four questions come up again and again.
Voice
Which speaker do you want? The OpenAI Text to speech guide says its endpoint offers 13 built-in voices, and that they are currently optimized for English. Pick one that fits your product and stick to it.
Speed and tone
How fast and in what mood? The same guide says one of its models lets you use prompts to control aspects such as tone and speed of speech.
Format
MP3 is that guide's default. It lists Opus for internet streaming and low latency, and WAV as uncompressed audio suited to low-latency uses.
Streaming
The guide says its speech endpoint supports realtime streaming using chunk transfer encoding, so "the audio can be played before the full file is generated".
Streaming matters because people hate waiting. Imagine a waiter who brings the whole meal at once, after twenty minutes of silence. Streaming is the waiter who brings the bread first. The first words play while the rest is still being made.
Here is a real request, copied from that guide. It is shown without its output and it is not run here, because it needs a key and costs money. Look at its shape: a model, the text, a voice and an optional instruction about tone.
# From the OpenAI Text to speech guide, shown without its output (not run here)
curl https://api.openai.com/v1/audio/speech \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"model": "gpt-4o-mini-tts",
"input": "Today is a wonderful day to build something people love!",
"voice": "coral",
"instructions": "Speak in a cheerful and positive tone."
}' \
--output speech.mp3Tip
Check the date on any docs page. The guide above says the models in this example are deprecated and will shut down on January 6, 2027. The shape of the request is the lesson here. The model names will change, so always read the current docs before you build.
Writing text for the ear
A page is read with the eyes. A voice is heard once, at a steady speed, and cannot be re-read. So text that looks fine on screen can sound bad out loud.
- 1
Keep sentences short
If a sentence needs a breath in the middle, split it. One idea per sentence.
- 2
Spell out numbers
Write "forty-two", not "42", when you want certainty. The Wikipedia page notes that "1325" could be read as "one thousand three hundred twenty-five", "one three two five", or as a year. The service has to guess.
- 3
Expand abbreviations
Write "Doctor", not "Dr.". And "St." could be "Street" or "Saint".
- 4
Watch tricky words
The same page gives "My latest project is to learn how to better project my voice", where "project" is said two ways. Words spelled alike but said differently are called heteronyms.
- 5
Listen to it
Always play the result. Your ears find problems your eyes miss.
Tip
Reading your text aloud to a friend is the cheapest test. Wherever you stumble, a voice will stumble too.
SSML: notes for the voice
Sometimes plain text is not enough. You want a pause here, or a word said a certain way. That is what SSML is for.
The W3C, the group that sets web standards, publishes the Speech Synthesis Markup Language specification. Its abstract says the language is meant "to provide authors of synthesizable content a standard way to control aspects of speech such as pronunciation, volume, pitch, rate, etc. across different synthesis-capable platforms".
It is XML, so the notes are tags in angle brackets, like <break/>, wrapped around your text. Here is an example from the spec.
<speak version="1.1" xml:lang="en-US">
Take a deep breath <break/> then continue.
Press 1 or wait for the tone. <break time="3s"/>
I didn't hear you! <break strength="weak"/> Please repeat.
</speak>Read it as a script with stage directions. The spec says the break element "controls the pausing or other prosodic boundaries between tokens". Prosody just means the rhythm and melody of speech. The time attribute sets a pause, here 3 seconds. The strength attribute has values from "none" and "x-weak" up to "x-strong", with "medium" as the default.
The spec also explains when to bother: the break element "is most often used to override the typical automatic behavior of a synthesis processor". In plain words, the voice already pauses in sensible places. Add a break only when it gets that wrong.
Tip
Not every service accepts every SSML tag. The spec gives a standard, but each service decides what it supports, so test a small sample before you rely on a tag.
Try it: a text normaliser and a beep
Now let's practise the preparing job in plain Python. No TTS model runs in this lesson. No voice will speak. A real model would be slower, would cost money, and could fail. Here we only do the part that is plain text work.
The program below does three things. It expands a few abbreviations. It turns numbers from 0 to 99 into words. It splits the text into short sentences. Then it wraps those sentences in a small SSML document with a 400-millisecond pause after each one. (A millisecond is a thousandth of a second.)
import re
ABBREVIATIONS = {r"\bDr\.": "Doctor", r"\bSt\.": "Street", r"\bapprox\.": "about", r"\bmin\b": "minutes"}
ONES = ["zero", "one", "two", "three", "four", "five", "six", "seven", "eight", "nine",
"ten", "eleven", "twelve", "thirteen", "fourteen", "fifteen", "sixteen",
"seventeen", "eighteen", "nineteen"]
TENS = ["", "", "twenty", "thirty", "forty", "fifty", "sixty", "seventy", "eighty", "ninety"]
def number_to_words(n):
if n < 20:
return ONES[n]
word = TENS[n // 10]
return word if n % 10 == 0 else word + "-" + ONES[n % 10]
def normalise(text):
for pattern, full in ABBREVIATIONS.items():
text = re.sub(pattern, full, text)
text = re.sub(r"\b\d{1,2}\b", lambda m: number_to_words(int(m.group())), text)
sentences = re.split(r"(?<=[.!?])\s+", text.strip())
return [s for s in sentences if s]
def build_ssml(sentences, pause_ms=400):
parts = [f'<s>{s}</s><break time="{pause_ms}ms"/>' for s in sentences]
return '<speak version="1.1" xml:lang="en-US">' + "".join(parts) + "</speak>"
raw = "Dr. Rao lives at 42 Mill St. near the park. The trip takes approx. 15 min by bus. Be there by 9!"
sentences = normalise(raw)
for s in sentences:
print(s)
print(build_ssml(sentences))Output
Doctor Rao lives at forty-two Mill Street near the park. The trip takes about fifteen minutes by bus. Be there by nine! <speak version="1.1" xml:lang="en-US"><s>Doctor Rao lives at forty-two Mill Street near the park.</s><break time="400ms"/><s>The trip takes about fifteen minutes by bus.</s><break time="400ms"/><s>Be there by nine!</s><break time="400ms"/></speak>
Read the output. The first three lines are the clean sentences. "42" became "forty-two", "15" became "fifteen" and "9" became "nine". The last line is the SSML, with an <s> tag around each sentence and a break after it.
The program is tiny on purpose. Real text is messier. It does not know that "St." can also mean Saint, and numbers above 99 are left alone. A real normaliser, or the service itself, handles many more cases.
What audio data is
When a service sends you speech, what is inside the file? Not words. It is a long list of numbers. A microphone measures the air pressure many times each second, and each measurement is one sample. Play the samples back in order and a speaker moves the air the same way.
To see this, we will make sound ourselves. A pure tone at 440 Hz (440 wiggles per second, the note musicians call A) is a sine wave, a smooth up-and-down curve. We measure it 16,000 times a second for half a second, and save it as a WAV file. Python's wave module, per its docs, supports uncompressed PCM WAV files. And if you give it a file-like object, it writes there instead of to disk. We use one that lives in memory, so nothing is saved.
import io
import math
import struct
import wave
RATE = 16000
SECONDS = 0.5
FREQ = 440
samples = []
for i in range(int(RATE * SECONDS)):
value = math.sin(2 * math.pi * FREQ * i / RATE)
samples.append(int(value * 32767 * 0.5))
print("samples:", len(samples))
print("first ten:", samples[:10])
buffer = io.BytesIO()
with wave.open(buffer, "wb") as wav:
wav.setnchannels(1)
wav.setsampwidth(2)
wav.setframerate(RATE)
wav.writeframes(struct.pack("<" + "h" * len(samples), *samples))
data = buffer.getvalue()
print("WAV bytes:", len(data))
print("starts with:", data[:4], data[8:12])Output
samples: 8000 first ten: [0, 2816, 5549, 8117, 10443, 12458, 14101, 15325, 16093, 16381] WAV bytes: 16044 starts with: b'RIFF' b'WAVE'
Half a second at 16,000 samples a second is 8,000 samples, which matches the first line. Each sample takes 2 bytes, so the sound itself is 16,000 bytes. The WAV file adds a 44-byte header at the front that says how to play it, so the total is 16,044 bytes. The last line shows the file begins with the letters RIFF and WAVE, the label that tells a player what it holds.
A real voice is the same kind of list, just much more complicated, and it is the back-end's job to compute it from your text.
Where it helps people
Text-to-speech is not only a feature. For many people it is the way they use a computer. The Wikipedia page says it has long been a vital assistive technology and notes that "The longest application has been in the use of screen readers for people with visual impairment, but text-to-speech systems are now commonly used by people with dyslexia and other reading disabilities as well as by pre-literate children."
That gives you real design jobs. Listen to your own app with a screen reader. Make sure the text is meaningful when spoken, not just when seen. Offer "read this aloud" on long pages. And remember that the same page notes high-speed synthesized speech is used by the visually impaired to quickly navigate computers, so do not clutter your text with noise that is only there for the eye.
Two rules for builders
A cloned or synthetic voice can be used to trick people. So two rules go with every project.
Say it is an AI voice
The OpenAI guide says its usage policies "require you to provide a clear disclosure to end users that the TTS voice they are hearing is AI-generated and not a human voice." Even where no policy forces you, tell your users. It is honest and it builds trust.
Never copy a real voice without consent
Do not make a voice that sounds like a real person unless that person has agreed. The same guide describes creating an approved custom voice "from a speaker's consent recording and matching audio sample". Consent is built into the process, and your own work should follow the same idea.
Read the provider's rules
Each service has its own policy page. Read it before you launch, because rules differ and change.
The Wikipedia page backs up why this matters. It records a case where a journalist recorded five minutes of himself talking and used a tool to create voice deepfakes that defeated a bank's voice-authentication system. A voice is not proof of who is speaking.
Quick check
Your app reads out a bill that says "Flat 7, 9 Oak St." The voice stumbles. What is the best first fix?
Quick check
A user is waiting for a long answer to be read out. Which setting helps the first words arrive sooner?
Recap
A TTS system has a front-end that prepares the text and a back-end that makes the sound. You choose a voice, a speed or tone, an audio format, and whether to stream. You write for the ear: short sentences, numbers and abbreviations in words. SSML, a W3C standard, lets you add pauses and pronunciation notes. And you disclose AI voices and never clone a real person's voice without consent. No model ran in this lesson, but you built the text work that comes before one.
Next: frameworks for multimodal apps.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.