Day 6 of 1060 minShell Walker
Day 6: Voice Cloning Detection with Spectrograms
Taught by Mr Elite Updated
By the end of today you can
- Extract and prepare an audio stream from a video with ffmpeg
- Generate and read spectrograms in Audacity and librosa
- Compare genuine and synthetic speech for breathing, pacing, noise floor and cut-offs
- Explain why phone-line audio weakens detection and rate each signal honestly
Warm-up
Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.
Why did we extract video frames as PNG rather than JPEG on Day 5?
PNG is lossless, so it adds no new compression artefacts that could be mistaken for manipulation.
What is the difference between what ffprobe tells you and what ffmpeg lets you study?
ffprobe reports what the file claims about its streams and codecs; ffmpeg lets you extract and inspect what the streams actually contain.
A finance officer gets a voicemail that sounds exactly like the managing director, asking for an urgent payment. The voice is calm, the accent is right, and the request sounds plausible. Your job is to say whether the audio is genuine, and to say it in a way you can defend.
Today I will show you that audio can be examined just like frames. You will not get a magic yes or no. You will learn to look at a sound as a picture, spot the habits of synthetic speech, and be honest about where the evidence runs out.
How synthetic voices are made, in plain terms
There are two common ways to make a fake voice. Text to speech (TTS) turns typed words into audio. Voice conversion takes a real person speaking and changes it to sound like someone else. Cloning tools train a model on samples of the target so either method sounds like them.
Most modern systems work in two stages. A first model predicts a compact picture of the sound from the text. A second model, called a vocoder, turns that picture into a waveform. Each stage makes small mistakes, and those mistakes are what we hunt for.
The first stage tends to get rhythm and emphasis slightly wrong, because it averages over many speakers it has heard. The second stage can leave a faint buzz or an unnaturally clean sound, because it never recorded a real room. Keep those two ideas in mind. Pacing problems point at the first stage, texture problems point at the second.
- Spectrogram
- A picture of sound. Time runs left to right, pitch runs bottom to top, and brightness shows how loud each pitch is.
- Vocoder
- The part of a synthetic voice system that turns a predicted sound picture into actual audio.
- Noise floor
- The faint background sound present in any real recording, such as room hum or air movement.
- Sample rate
- How many audio measurements are stored per second. 16000 Hz means 16000 per second.
Reading a spectrogram and the clues to look for
First, learn the picture. The horizontal axis is time in seconds. The vertical axis is frequency in Hz, low voice tones at the bottom and hissy sounds like s and f near the top. Bright colour means loud. Vowels appear as stacked horizontal stripes, called harmonics. Consonants appear as short bursts or fuzzy patches.
Once you can read it, four things are worth checking. I rate each one separately, as we did with video. For each, I note what I saw, an innocent explanation, and a strength of weak, moderate or strong.
Quick check
A clip has perfectly silent gaps between sentences and no breaths. What is the fairest conclusion?
Silence is consistent with synthesis but also with innocent processing, so it is a weak signal to be combined with others.
Watch me do it: build a real and a synthetic pair
We will make our own evidence, so we know the truth. Everything runs on your own machine with your own voice, and no third party is involved. I assume ffmpeg from earlier days is installed. If a command is not found, install it before going on.
sudo apt install -y espeak-ng
espeak-ng --version
mkdir -p day06 && cd day06espeak-ng -w synth.wav "Please confirm the payment before the end of the day."ffmpeg -i real.wav -ac 1 -ar 16000 real16.wav
ffmpeg -i synth.wav -ac 1 -ar 16000 synth16.wav
ffprobe -hide_banner real16.wavQuick check
Why do we convert both files to the same sample rate and channels before comparing?
Matching format removes a variable, so any difference is more likely about the sound itself.
Spectrograms in Audacity and librosa
Now we look at them. Start with Audacity because it is quick and visual, then use librosa (a Python audio library) so the result is saved as a file you can put in a report.
import librosa, librosa.display
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
for name in ['real16', 'synth16']:
y, sr = librosa.load(name + '.wav', sr=16000)
S = librosa.amplitude_to_db(np.abs(librosa.stft(y)), ref=np.max)
fig, ax = plt.subplots(figsize=(10, 4))
img = librosa.display.specshow(S, sr=sr, x_axis='time', y_axis='hz', ax=ax)
fig.colorbar(img, ax=ax, format='%+2.0f dB')
ax.set_title(name)
fig.savefig(name + '.png', dpi=120)
print('saved', name + '.png')Look at your two images. The espeak-ng one is a very obvious synthetic voice, with sharp, regular bands and almost no noise between words. Yours will show a speckled floor and soft breath marks. Modern cloning tools are far more convincing than espeak-ng, so treat this as training for your eye, not as a measure of real-world difficulty.
Pictures are good, but a number is easier to put in a report. This second script estimates the noise floor by taking the loudness of the quietest tenth of the clip. It is crude, yet it gives you something to quote.
import librosa, numpy as np
for name in ['real16', 'synth16']:
y, sr = librosa.load(name + '.wav', sr=16000)
rms = librosa.feature.rms(y=y)[0]
quiet = np.percentile(rms, 10)
print(name, 'noise floor (rms 10th pct):', round(float(quiet), 5))Save it as noise.py and run python3 noise.py. Expect two lines. Your real file should show a small but clearly non-zero value, often in the thousandths. The espeak-ng file will usually print 0.0 or something very close. Exact numbers depend on your microphone and room, so the comparison matters, not the figure.
Why phone-line audio makes this harder
Phone calls and voicemail squeeze audio hard. Narrowband telephony keeps roughly 300 to 3400 Hz and throws away the rest. Codecs also smooth out fine detail, add their own artefacts and fill silence with generated comfort noise.
That matters because many clues live in the detail that phones remove. A real recording gets a hard cut-off at 3400 Hz, and so does a fake that went through the same line. The two start to look alike. Detection tools trained on clean audio also lose accuracy on compressed audio.
You can see this yourself. The command below applies a phone-style band filter to both of your files. Then rerun spec.py with the names changed to real_phone and synth_phone, or open the new files in Audacity.
ffmpeg -i real16.wav -af "highpass=f=300,lowpass=f=3400" real_phone.wav
ffmpeg -i synth16.wav -af "highpass=f=300,lowpass=f=3400" synth_phone.wavIn the spectrogram you should see everything above 3400 Hz turn black on both tracks, with the stripes below that looking much more alike than before. That is the point. A clue you could see in clean audio is now partly hidden, and your confidence must drop with it.
Quick check
A voicemail shows a hard cut-off above 3.4 kHz. What does this most likely indicate?
Phone systems remove those frequencies from all callers, real or synthetic.
If it did not work
Comparing files with different sample rates or channel counts
Convert both with ffmpeg -ac 1 -ar 16000 first, and confirm with ffprobe.
Treating espeak-ng results as proof of how modern clones look
Remember it is a simple synthesiser. Use it to learn the method, and expect real clones to be subtler.
Declaring a clip fake from a single clue
Rate each signal separately, note an innocent explanation, and look for agreement between independent signals.
Ignoring the original recording path
Ask whether the audio came from a call, a messaging app or a direct file, since each compresses differently.
Recap
- Synthetic speech is usually predicted as a sound picture, then turned into audio by a vocoder, and each stage can leave traces
- A spectrogram turns audio into a picture you can compare by eye and save as evidence
- Check breathing, pacing, background noise and cut-offs one at a time
- Match sample rate and channels before comparing two files
- Phone and voicemail compression removes or mimics many clues, so confidence must fall
- No single audio signal proves a fake; look for agreement and state your uncertainty
Cheat sheet
ffmpeg -i in.mp4 -vn -ac 1 -ar 16000 out.wav- Extract the audio as mono 16 kHz WAV
espeak-ng -w out.wav "text"- Generate a synthetic speech file
ffprobe -hide_banner file.wav- Show codec, sample rate and channels
Audacity track menu, Spectrogram- Switch a track to spectrogram view
librosa.stft- Computes the data behind a spectrogram
ffmpeg -af "highpass=f=300,lowpass=f=3400"- Imitate a phone line to test how clues fade
Noise floor- Background level in the quiet gaps; near zero is suspicious but not proof
Narrowband telephony- About 300 to 3400 Hz, which hides fine detail
Before the next day
Mission: Real versus synthetic spectrogram pair
Record yourself saying a sentence, generate the same sentence with espeak-ng, and compare the two as an analyst would. This builds the habit of testing your method on material where you already know the truth.
About 30 min
You are done when: A day06 folder containing real.wav, synth.wav, two spectrogram PNGs and a notes06.md with rated observations.
Want more? Play your real.wav through a phone speaker and re-record it, or compress both files to low-bitrate MP3 with ffmpeg, then redo the comparison and note which clues were lost.
Quiz
4 questions. Your score is saved with your progress.
1. What does a vocoder do in a synthetic voice system?
The vocoder is the stage that produces the actual audio from the model's predicted sound picture.
2. Two clips have different noise floors, but one was recorded as a WAV and the other was saved as a low-bitrate MP3. What should you do first?
Format and compression change noise floor, so you must remove that variable before reading meaning into the difference.
3. Why is a hard cut-off at 3.4 kHz weak evidence of cloning?
Band-limiting is an ordinary property of telephony, so genuine and synthetic calls both show it.
4. Which report statement is the most defensible?
It states what was observed, offers an innocent explanation and sets the strength of the signal.
Finish the day
Mark the day complete to save your progress.
Next: Day 7 is an apply-it day. You will take a video with audio and run everything from Days 1 to 6 on it, then look at how detectors can be evaded and why that limits what you can claim.
Your notes
Private to this device. Write what you want to remember from today.