Speech Recognition & Audio ML

By the end of this lesson you'll understand the whole pipeline that turns spoken audio into text — from raw waveforms to features to neural models — and you'll be able to decode model output and score it with Word Error Rate in plain Python.

Learn Speech Recognition & Audio ML in our free AI & Machine Learning course — a beginner-friendly interactive lesson with worked examples, a practice…

Part of the free AI & Machine Learning course at LearnCodingFast — hands-on lessons with examples you run in your browser, plus practice exercises and a quick quiz.

Think of speech recognition like a musician transcribing a song by ear. First the air pressure wiggles — that's the raw sound wave . A microphone measures that wiggle thousands of times a second, giving a long list of numbers (the waveform ).

The musician doesn't read raw pressure values — they "see" the music as pitches over time. The computer does the same by drawing a spectrogram : a picture of which frequencies are loud at each moment, like sheet music for sound. Then a trained "reader" (the neural network) looks at that picture and writes down the words.

The hard part is the same one a human transcriber faces: people speak at different speeds, with accents, mumbling, and a noisy café in the background. Good ASR is mostly about reading that messy, smudged sheet music correctly.

Sound is a wave of changing air pressure. To get it into a computer, a microphone samples that wave — it measures the pressure at fixed instants and writes down a number each time. Do that 16,000 times a second and you have a 16kHz recording: a list of 16,000 numbers per second.

That number (16,000) is the sample rate . Higher sample rates capture higher frequencies but make bigger files. Speech lives mostly below 8kHz, so almost every speech model expects 16kHz mono audio — a single channel sampled 16,000 times a second.

Before any neural network, you can already pull useful signals straight from the raw samples. Energy (the sum of squared samples) tells you how loud a slice is. Zero-crossings (how often the wave flips between positive and negative) hint at pitch and noisiness. Run the example below to compute both on a toy 12-sample "waveform".

Raw samples are hard for a network to read directly — there are too many, and they don't show pitch clearly. So you transform them into features : a compact picture of the sound.

A spectrogram chops the audio into short overlapping frames (say 25ms every 10ms) and, for each frame, measures how much energy sits at each frequency. Stack those frames side by side and you get a 2D image: time across the bottom, frequency up the side, brightness = loudness.

A mel spectrogram warps the frequency axis to match human hearing — we hear low pitches in fine detail and high pitches coarsely, so the mel scale packs more resolution into the low and mid range where speech lives. MFCCs (Mel-Frequency Cepstral Coefficients) go one step further: they compress each mel frame into ~13 numbers that summarise the shape of the sound. MFCCs powered classic ASR for decades; modern deep models often feed on the mel spectrogram directly.

Waveform → frame it → FFT per frame → mel scale → log → (optional) MFCC

The result is a 2D array, e.g. (80 mel bands × N time frames) . Speech models treat this like a single-channel image.

Classic recognition splits the work in two. The acoustic model answers "what sounds are in this audio?" — it maps spectrogram frames to phonemes or characters. It hears sounds , not meaning, so on its own it happily produces "wreck a nice beach".

The language model answers "which word sequence is actually likely?" — it knows that "recognise speech" is far more probable than "wreck a nice beach". It rescues the acoustic model from sound-alike mistakes (homophones) and rare words.

Modern end-to-end models often fold both jobs into one network, but the language-model idea still shows up — for example, Whisper's text decoder has implicitly learned which word sequences are plausible.

Here's the alignment problem: a one-second clip might be 100 frames, but the word "hi" is 2 letters. Which frames belong to "h" and which to "i"? Hand-labelling that for millions of clips is impossible.

CTC (Connectionist Temporal Classification) solves it cleverly. The model emits one symbol per frame and is allowed to (a) repeat a symbol and (b) emit a special blank meaning "no new letter here". To read the final text you collapse runs of the same symbol, then delete the blanks. So h-e-l-l-o becomes hello . During training, CTC sums the probability of every frame-alignment that collapses to the correct text — so the model never needs a human to say where each letter starts.

The clever bit: double letters survive because the model puts a blank between them. In h-e-l-l-o the two l's are kept; remove that blank and they would collapse to a single l, so you could never spell "hello". Run the decoder below to see both rules in action.

Today's best systems are end-to-end : a single neural network goes straight from audio (or mel spectrogram) to text, learning the features and the alignment itself.

You don't implement these from scratch. Here's the full Whisper "hello world" — read it as a worked example (you can't run model downloads in the sandbox, so the expected output is shown in comments):

The Hugging Face Transformers version, which works for Whisper and wav2vec 2.0 alike:

How good is a transcription? The industry metric is Word Error Rate . You line up the model's words against the truth and count the cheapest set of edits to fix them — substitutions , insertions , and deletions — then divide by the number of words in the reference:

Lower is better. 0.0 is perfect; a WER of 0.25 means one word in four is wrong. That "cheapest set of edits" is exactly the classic edit distance (Levenshtein) — but counted on whole words instead of characters. The example below computes it from scratch.

ASR turns audio into text; text-to-speech runs the pipeline in reverse — text in, audio out. A model like Tacotron or VITS first generates a mel spectrogram from the words, then a vocoder (such as HiFi-GAN) turns that spectrogram back into a waveform you can play.

Notice the symmetry: the mel spectrogram sits in the middle of both tasks. It's the common "sheet music" representation that bridges sound and meaning in either direction.

Put it together with no scaffolding. Decode a raw CTC string into a command, then score it against the expected command with WER. The starter only gives you an outline — write the logic yourself.

These four trip up almost everyone building their first speech system:

Background noise, music, or two people talking smear the spectrogram and the model guesses wrong.

✅ Fix: denoise or apply Voice Activity Detection (VAD) to keep only speech, and prefer a model trained on noisy data (Whisper handles noise well). Recording closer to the mic helps more than any code.

Feeding 44.1kHz audio to a model trained on 16kHz (or vice versa) shifts every frequency and produces nonsense.

✅ Fix: always resample to the model's expected rate and convert to mono before inference.

A model trained mostly on US English mangles strong regional accents, names, brand names, and jargon it never saw.

✅ Fix: choose a multilingual / multi-accent model (Whisper), fine-tune on samples from your domain, or bias decoding toward a custom vocabulary of expected terms.

A pure acoustic model with no sense of likely word sequences writes "to too two" interchangeably and invents non-words.

✅ Fix: add or enable a language model (an n-gram LM with CTC beam search, or use a seq2seq model like Whisper whose decoder already knows likely word sequences).

Lesson complete — you can read the whole ASR pipeline!

You now know how audio becomes numbers, how spectrograms and MFCCs turn it into features a network can read, how acoustic and language models divide the work, why CTC lets models learn alignment for free, and how to use Whisper and wav2vec 2.0. You can even decode CTC output and score a transcription with WER by hand.

🚀 Up next: Advanced NLP — transformers for understanding and generating text, the same architecture that powers Whisper's decoder.

Practice quiz

• Browse all free courses
• Cheat sheets & quizzes
• Coding dictionary