AI engineering / Advanced / 03.29
Turning speech into text
How an app turns a recording into words, how to score the result, and how to prepare audio so a model can read it.
14 min readFree
From a recording to words
The last lesson, Image generation, ended with a promise: next, speech-to-text, turning recordings into words. Here it is.
Think of a friend who types out everything said in a meeting. Speech-to-text does the same job, with a model as the typist. You give it sound. You get text back.
This matters when you build products. Text is easy to search, to translate, to summarise and to hand to a chat model. A voice note, a call or a lecture is none of those until it becomes words.
By the end you will know what can go wrong, how to measure a transcript with one number, what timestamps and speaker labels are, and how to prepare a long file. You will run two small Python programs. No speech model runs in this lesson. The code teaches the ideas around a model.
Key words
- Speech-to-text
- Software that listens to a recording and writes down the words. Also called speech recognition or transcription.
- Transcript
- The text the software gives back.
- Reference
- The correct text, written by a person. You compare a transcript to it.
- Word error rate (WER)
- A score for how far a transcript is from the reference. Lower is better.
- Timestamp
- A time label, like "this word starts at 4.2 seconds".
- Diarization
- Working out who spoke when, so each line gets a speaker label.
- Sample rate
- How many times per second a recording measures the sound.
- Mono
- One audio channel. Stereo has two, left and right.
What makes it hard
People are not studio microphones. A clear voice in a quiet room is the easy case. Real recordings are messier, and each mess is a place your product can slip.
Noise
A busy street, a fan, a clattering plate. Like a loud party: the words are there, but hard to pick out.
Accents and voices
People pronounce the same word in many ways, and speak fast or slow, loud or soft. A model may handle some voices better than others.
Your own words
Brand names, product codes and medical terms are rare in everyday speech, so they are easy to mishear.
This lesson gives no numbers for these three. What it does give you is a way to find out for yourself: record real samples, write the correct text, and score the transcript. That score is the next section.
Tip
Test with the voices and rooms your users have, not only a clean demo clip. A model that does well on a quiet recording tells you little about a noisy kitchen.
Scoring a transcript
Say a model hears "turn the light off in the kitchen" when the person said "turn the lights off in the kitchen". Is that good or bad? One word is wrong out of seven. You want that said as a number, so you can compare two models or two versions of your own setup.
The usual number is the word error rate. The Wikipedia page on it says the measure is derived from the Levenshtein distance, "working at the word level instead of the phoneme level". In plain words: count the fewest edits needed to turn the reference into the transcript, then divide by how many words the reference has.
The page names three kinds of edit. A substitution swaps one word for another. A deletion drops a word. An insertion adds a word that was never said. The formula is below.
In words: add up the substitutions (S), the deletions (D) and the insertions (I), then divide by N, the number of words in the reference. Zero means a perfect copy. Because insertions count, the page notes the score can go above 1.0, if there are more insertions than correct words.
Tip
WER tells you how much is wrong, not which words. Read some transcripts yourself too.
Computing WER by hand
Finding the fewest edits is a small puzzle. We solve it with a table. Each cell holds the fewest edits to turn the first few reference words into the first few heard words. Fill the table from the top left to the bottom right. The bottom right cell is the answer.
- 1
Start the edges
Turning i words into nothing takes i deletions. Turning nothing into j words takes j insertions. That fills the first row and column.
- 2
Fill each cell
Take the cheapest of three moves: drop a word (the cell above plus 1), add a word (the cell to the left plus 1), or compare two words (the cell diagonally up-left, plus 1 only if the words differ).
- 3
Read the corner
The bottom right cell is the number of edits. Divide it by the reference length.
Here it is in plain Python. We score two made-up transcripts of the same sentence. No model made them. I typed them.
def edit_table(ref, hyp):
# table[i][j] = fewest edits to turn the first i reference words into the first j heard words
table = [[0] * (len(hyp) + 1) for _ in range(len(ref) + 1)]
for i in range(len(ref) + 1):
table[i][0] = i
for j in range(len(hyp) + 1):
table[0][j] = j
for i in range(1, len(ref) + 1):
for j in range(1, len(hyp) + 1):
same = 0 if ref[i - 1] == hyp[j - 1] else 1
table[i][j] = min(table[i - 1][j] + 1, # a word was dropped
table[i][j - 1] + 1, # an extra word appeared
table[i - 1][j - 1] + same) # same word, or swapped
return table
reference = "turn the lights off in the kitchen".split()
transcripts = {
"A": "turn the light off in the kitchen".split(),
"B": "turn lights off in the big kitchen please".split(),
}
for name, heard in transcripts.items():
table = edit_table(reference, heard)
edits = table[-1][-1]
print("Transcript", name, ":", " ".join(heard))
print(" edits:", edits, "reference words:", len(reference))
print(" WER:", round(edits / len(reference), 3))
print()
print("Table for transcript A (rows = reference words):")
for row in edit_table(reference, transcripts["A"]):
print(row)Output
Transcript A : turn the light off in the kitchen edits: 1 reference words: 7 WER: 0.143 Transcript B : turn lights off in the big kitchen please edits: 3 reference words: 7 WER: 0.429 Table for transcript A (rows = reference words): [0, 1, 2, 3, 4, 5, 6, 7] [1, 0, 1, 2, 3, 4, 5, 6] [2, 1, 0, 1, 2, 3, 4, 5] [3, 2, 1, 1, 2, 3, 4, 5] [4, 3, 2, 2, 1, 2, 3, 4] [5, 4, 3, 3, 2, 1, 2, 3] [6, 5, 4, 4, 3, 2, 1, 2] [7, 6, 5, 5, 4, 3, 2, 1]
Transcript A swapped "lights" for "light". That is one edit out of seven reference words, so WER is 0.143. Transcript B dropped "the" before "lights", added "big" and added "please". That is three edits, so WER is 0.429.
In the table for A, the numbers on the diagonal stay at 0 for the first two words, then rise to 1 at "lights" and stay there. That is one edit, and the bottom right cell says so.
Quick check
A reference has 10 words. The transcript needs 2 substitutions, 1 deletion and 2 insertions to match it. What is the WER?
Timestamps and speaker labels
A plain block of text is rarely enough. If you build captions, or a way to jump to the moment someone said something, you need to know when each word was said. That is what timestamps are for.
The OpenAI docs describe a timestamp option on their transcription endpoint. It returns structured timestamp data that the page says is for captioning and video editing. Docs change, so read the current page before you build.
The second kind of label answers a different question: who is talking? A meeting transcript with no names is hard to follow. Wikipedia calls this speaker diarisation (or diarization). It is "the process of partitioning an audio stream containing human speech into homogeneous segments according to the identity of each speaker". It is used to answer the question "who spoke when?"
The page splits the job in two. Speaker segmentation finds the points where the speaker changes. Speaker clustering groups the pieces that sound like the same person. Notice what diarization gives you: Speaker 1, Speaker 2. To get a real name, the page says it must be used together with speaker recognition systems.
Words only
Plain text. Fine for search, summaries and notes.
Words with times
Each word or sentence gets a start time. Needed for captions and for jumping around a recording.
Words, times and speakers
Each line also says who spoke. Needed for meetings, interviews and call reviews.
The Whisper model
Whisper is a speech model described in a research paper called Robust Speech Recognition via Large-Scale Weak Supervision. Here is its abstract, word for word.
"We study the capabilities of speech processing systems trained simply to predict large amounts of transcripts of audio on the internet. When scaled to 680,000 hours of multilingual and multitask supervision, the resulting models generalize well to standard benchmarks and are often competitive with prior fully supervised results but in a zero-shot transfer setting without the need for any fine-tuning. When compared to humans, the models approach their accuracy and robustness. We are releasing models and inference code to serve as a foundation for further work on robust speech processing."
Three plain-word points. First, it learned from a huge pile of audio paired with transcripts: 680,000 hours. Second, the abstract says the results come "without the need for any fine-tuning", which means no extra training on each test. Third, the authors released the models and code. The Fine-tuning lesson goes deeper on retraining a model.
Calling a transcription endpoint
Most teams do not run a speech model on their own machines. They send the audio file to a web address and get text back. The OpenAI File transcription guide shows the shortest version in Python. It is copied below exactly as the page shows it. It was not run here. It needs a paid account and a key, so we will not run it.
# From the File transcription guide in the OpenAI API docs, shown without its output (not run here)
from openai import OpenAI
client = OpenAI()
audio_file = open("audio.wav", "rb")
transcription = client.audio.transcriptions.create(
model="gpt-transcribe",
file=audio_file
)
print(transcription.text)Read it plainly: load the library, make a client, open a local audio file, send it with a model name, print the text that comes back.
This is a real service, so remember what a stand-in would hide. It is slower than a local function, it costs money on every call, it can return slightly different text on a second run, and it can fail or refuse.
The same guide has an example that passes a short prompt describing the call and a list of keywords, with words like a plan name, an account code and a billing term. That is one way to hand a model the rare words of your own field. The guide does not give a score for how much it helps, so measure it with the WER code above.
The guide also says what the endpoint takes. Files can be up to 25 MB. Supported input formats include flac, mp3, mp4, mpeg, mpga, m4a, ogg, wav and webm. The docs also mark the Whisper model name, whisper-1, as deprecated, with a shutdown on February 26, 2027.
A file or a live stream
There are two ways to feed sound in. The docs split them like this: use file transcription "when you have a completed recording or a bounded audio request". For audio still arriving from a microphone, a call or a media stream, they point to a separate realtime transcription feature.
File
Wait until the recording ends, send it all, get one final transcript. Like handing a tape to a typist. Simpler, fine for voice notes and uploads.
Live stream
Send small pieces as they are spoken and get words back as you go. Like captions on live TV. Needed for voice assistants and live captions.
The trade
Live feels fast but is harder to build. A file is simple, but you wait for the whole recording.
Quick check
You are building a feature where users upload a finished voice memo. Which fits best?
Preparing audio
Before you send a recording, look at what is in it. A microphone measures the air many times a second, and each measurement is a number. Three settings decide how big the file is.
Key words
- Sample
- One measurement of the sound at one instant.
- Sample width
- How many bytes one measurement takes. Two bytes is common.
- Frame
- One sample from every channel at the same instant.
The sample rate is how many frames you get each second. Python's wave module, which reads and writes WAV files, has calls to read the sample rate, the channel count and the sample width. The Python docs say its channel call returns 1 for mono and 2 for stereo.
The program below writes a tiny WAV file in memory: one second of silence, 16,000 frames, mono, two bytes a sample. Then it reads the file back and prints what it finds. Again, there is no speech in it, and no model.
import io
import wave
# Write 1 second of silence: 16,000 frames of two zero bytes each (16-bit), one channel.
buffer = io.BytesIO()
with wave.open(buffer, "wb") as writer:
writer.setnchannels(1)
writer.setsampwidth(2)
writer.setframerate(16000)
writer.writeframes(b"\x00\x00" * 16000)
buffer.seek(0)
with wave.open(buffer, "rb") as reader:
rate = reader.getframerate()
channels = reader.getnchannels()
frames = reader.getnframes()
print("sample rate:", rate, "per second")
print("channels:", channels)
print("sample width:", reader.getsampwidth(), "bytes")
print("frames:", frames)
print("duration:", frames / rate, "seconds")
print("bytes of sound data:", 16000 * 2 * 1)
print("bytes in 10 minutes:", 16000 * 2 * 1 * 600)Output
sample rate: 16000 per second channels: 1 sample width: 2 bytes frames: 16000 duration: 1.0 seconds bytes of sound data: 32000 bytes in 10 minutes: 19200000
16,000 frames at 16,000 a second is 1.0 seconds, as it should be. Each frame is 2 bytes, so one second is 32,000 bytes. Ten minutes is 600 seconds, which gives 19,200,000 bytes, under the 25 MB limit. Those numbers are for this exact format, so check your own recordings.
Long files
What if the recording is bigger than the limit? The docs say that for larger recordings you should use a compressed audio format or split the file into chunks of 25 MB or less. They add: avoid splitting in the middle of a sentence, because that can remove context and reduce accuracy.
- 1
Try compressing
The docs name compressed formats as one way out.
- 2
Cut at pauses
If it is still too big, cut it into chunks where nobody is speaking, not mid-sentence.
- 3
Stitch the text
Transcribe each chunk, then join the pieces in order. If you want timestamps, add each chunk's start time to its times.
Tip
One channel takes half the bytes of two. If your recordings are stereo, test whether mono still scores well on your own samples.
Quick check
A call recording is over the size limit. What do the docs suggest?
What to remember
Test on real audio
Noise, accents and unusual words are the places a transcript slips. Score your own samples.
Score with WER
(S + D + I) / N, the edits needed divided by the reference words. Lower is better.
Match the tool to the job
Words only, words with times, or words with speakers. A file for finished audio, a live stream for audio still arriving.
Once you have text, a chat model counts it in tokens. The Tokenization lesson in Machine Learning Foundations goes deeper on that.
Next: text-to-speech, giving your app a voice.
Sources
Last updated on 16 November 2026
Learn with me
Every lesson is free. I add new ones as I learn. Have a question, or want help with your own work? Book a short call.