Day 6 of 1060 minShell Walker

Day 6: Voice Cloning Detection with Spectrograms

Taught by Mr Elite Updated

By the end of today you can

  • Extract and prepare an audio stream from a video with ffmpeg
  • Generate and read spectrograms in Audacity and librosa
  • Compare genuine and synthetic speech for breathing, pacing, noise floor and cut-offs
  • Explain why phone-line audio weakens detection and rate each signal honestly

Warm-up

Answer from memory first, then tap to check. Recalling yesterday is what makes it stick.

Why did we extract video frames as PNG rather than JPEG on Day 5?

PNG is lossless, so it adds no new compression artefacts that could be mistaken for manipulation.

What is the difference between what ffprobe tells you and what ffmpeg lets you study?

ffprobe reports what the file claims about its streams and codecs; ffmpeg lets you extract and inspect what the streams actually contain.

A finance officer gets a voicemail that sounds exactly like the managing director, asking for an urgent payment. The voice is calm, the accent is right, and the request sounds plausible. Your job is to say whether the audio is genuine, and to say it in a way you can defend.

Today I will show you that audio can be examined just like frames. You will not get a magic yes or no. You will learn to look at a sound as a picture, spot the habits of synthetic speech, and be honest about where the evidence runs out.

How synthetic voices are made, in plain terms

There are two common ways to make a fake voice. Text to speech (TTS) turns typed words into audio. Voice conversion takes a real person speaking and changes it to sound like someone else. Cloning tools train a model on samples of the target so either method sounds like them.

Most modern systems work in two stages. A first model predicts a compact picture of the sound from the text. A second model, called a vocoder, turns that picture into a waveform. Each stage makes small mistakes, and those mistakes are what we hunt for.

The first stage tends to get rhythm and emphasis slightly wrong, because it averages over many speakers it has heard. The second stage can leave a faint buzz or an unnaturally clean sound, because it never recorded a real room. Keep those two ideas in mind. Pacing problems point at the first stage, texture problems point at the second.

Spectrogram
A picture of sound. Time runs left to right, pitch runs bottom to top, and brightness shows how loud each pitch is.
Vocoder
The part of a synthetic voice system that turns a predicted sound picture into actual audio.
Noise floor
The faint background sound present in any real recording, such as room hum or air movement.
Sample rate
How many audio measurements are stored per second. 16000 Hz means 16000 per second.

Reading a spectrogram and the clues to look for

First, learn the picture. The horizontal axis is time in seconds. The vertical axis is frequency in Hz, low voice tones at the bottom and hissy sounds like s and f near the top. Bright colour means loud. Vowels appear as stacked horizontal stripes, called harmonics. Consonants appear as short bursts or fuzzy patches.

Once you can read it, four things are worth checking. I rate each one separately, as we did with video. For each, I note what I saw, an innocent explanation, and a strength of weak, moderate or strong.

Quick check

A clip has perfectly silent gaps between sentences and no breaths. What is the fairest conclusion?

Watch me do it: build a real and a synthetic pair

We will make our own evidence, so we know the truth. Everything runs on your own machine with your own voice, and no third party is involved. I assume ffmpeg from earlier days is installed. If a command is not found, install it before going on.

bash
sudo apt install -y espeak-ng
espeak-ng --version
mkdir -p day06 && cd day06
Install the tool, check it runs, and make a working folder.
bash
espeak-ng -w synth.wav "Please confirm the payment before the end of the day."
Writes the spoken sentence to synth.wav. Say the same sentence into Audacity for real.wav.
bash
ffmpeg -i real.wav -ac 1 -ar 16000 real16.wav
ffmpeg -i synth.wav -ac 1 -ar 16000 synth16.wav
ffprobe -hide_banner real16.wav
Converts both to mono 16 kHz. The last line lets you confirm the stream details.

Quick check

Why do we convert both files to the same sample rate and channels before comparing?

Spectrograms in Audacity and librosa

Now we look at them. Start with Audacity because it is quick and visual, then use librosa (a Python audio library) so the result is saved as a file you can put in a report.

python
import librosa, librosa.display
import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt

for name in ['real16', 'synth16']:
    y, sr = librosa.load(name + '.wav', sr=16000)
    S = librosa.amplitude_to_db(np.abs(librosa.stft(y)), ref=np.max)
    fig, ax = plt.subplots(figsize=(10, 4))
    img = librosa.display.specshow(S, sr=sr, x_axis='time', y_axis='hz', ax=ax)
    fig.colorbar(img, ax=ax, format='%+2.0f dB')
    ax.set_title(name)
    fig.savefig(name + '.png', dpi=120)
    print('saved', name + '.png')
Loads each file, computes a short-time Fourier transform (the sound picture), and saves it as a PNG.

Look at your two images. The espeak-ng one is a very obvious synthetic voice, with sharp, regular bands and almost no noise between words. Yours will show a speckled floor and soft breath marks. Modern cloning tools are far more convincing than espeak-ng, so treat this as training for your eye, not as a measure of real-world difficulty.

Pictures are good, but a number is easier to put in a report. This second script estimates the noise floor by taking the loudness of the quietest tenth of the clip. It is crude, yet it gives you something to quote.

python
import librosa, numpy as np
for name in ['real16', 'synth16']:
    y, sr = librosa.load(name + '.wav', sr=16000)
    rms = librosa.feature.rms(y=y)[0]
    quiet = np.percentile(rms, 10)
    print(name, 'noise floor (rms 10th pct):', round(float(quiet), 5))
Prints a rough noise floor figure. A value near zero means the gaps are almost digital silence.

Save it as noise.py and run python3 noise.py. Expect two lines. Your real file should show a small but clearly non-zero value, often in the thousandths. The espeak-ng file will usually print 0.0 or something very close. Exact numbers depend on your microphone and room, so the comparison matters, not the figure.

Why phone-line audio makes this harder

Phone calls and voicemail squeeze audio hard. Narrowband telephony keeps roughly 300 to 3400 Hz and throws away the rest. Codecs also smooth out fine detail, add their own artefacts and fill silence with generated comfort noise.

That matters because many clues live in the detail that phones remove. A real recording gets a hard cut-off at 3400 Hz, and so does a fake that went through the same line. The two start to look alike. Detection tools trained on clean audio also lose accuracy on compressed audio.

You can see this yourself. The command below applies a phone-style band filter to both of your files. Then rerun spec.py with the names changed to real_phone and synth_phone, or open the new files in Audacity.

bash
ffmpeg -i real16.wav -af "highpass=f=300,lowpass=f=3400" real_phone.wav
ffmpeg -i synth16.wav -af "highpass=f=300,lowpass=f=3400" synth_phone.wav
Keeps only 300 to 3400 Hz, imitating a phone line. Both files are your own.

In the spectrogram you should see everything above 3400 Hz turn black on both tracks, with the stripes below that looking much more alike than before. That is the point. A clue you could see in clean audio is now partly hidden, and your confidence must drop with it.

Quick check

A voicemail shows a hard cut-off above 3.4 kHz. What does this most likely indicate?

If it did not work

Comparing files with different sample rates or channel counts

Convert both with ffmpeg -ac 1 -ar 16000 first, and confirm with ffprobe.

Treating espeak-ng results as proof of how modern clones look

Remember it is a simple synthesiser. Use it to learn the method, and expect real clones to be subtler.

Declaring a clip fake from a single clue

Rate each signal separately, note an innocent explanation, and look for agreement between independent signals.

Ignoring the original recording path

Ask whether the audio came from a call, a messaging app or a direct file, since each compresses differently.

Recap

  • Synthetic speech is usually predicted as a sound picture, then turned into audio by a vocoder, and each stage can leave traces
  • A spectrogram turns audio into a picture you can compare by eye and save as evidence
  • Check breathing, pacing, background noise and cut-offs one at a time
  • Match sample rate and channels before comparing two files
  • Phone and voicemail compression removes or mimics many clues, so confidence must fall
  • No single audio signal proves a fake; look for agreement and state your uncertainty

Cheat sheet

ffmpeg -i in.mp4 -vn -ac 1 -ar 16000 out.wav
Extract the audio as mono 16 kHz WAV
espeak-ng -w out.wav "text"
Generate a synthetic speech file
ffprobe -hide_banner file.wav
Show codec, sample rate and channels
Audacity track menu, Spectrogram
Switch a track to spectrogram view
librosa.stft
Computes the data behind a spectrogram
ffmpeg -af "highpass=f=300,lowpass=f=3400"
Imitate a phone line to test how clues fade
Noise floor
Background level in the quiet gaps; near zero is suspicious but not proof
Narrowband telephony
About 300 to 3400 Hz, which hides fine detail

Before the next day

Mission: Real versus synthetic spectrogram pair

Record yourself saying a sentence, generate the same sentence with espeak-ng, and compare the two as an analyst would. This builds the habit of testing your method on material where you already know the truth.

About 30 min

You are done when: A day06 folder containing real.wav, synth.wav, two spectrogram PNGs and a notes06.md with rated observations.

Want more? Play your real.wav through a phone speaker and re-record it, or compress both files to low-bitrate MP3 with ffmpeg, then redo the comparison and note which clues were lost.

Quiz

4 questions. Your score is saved with your progress.

1. What does a vocoder do in a synthetic voice system?

2. Two clips have different noise floors, but one was recorded as a WAV and the other was saved as a low-bitrate MP3. What should you do first?

3. Why is a hard cut-off at 3.4 kHz weak evidence of cloning?

4. Which report statement is the most defensible?

Finish the day

Mark the day complete to save your progress.

Go to Day 7

Next: Day 7 is an apply-it day. You will take a video with audio and run everything from Days 1 to 6 on it, then look at how detectors can be evaded and why that limits what you can claim.

Tell someone you are doing this:WhatsAppXLinkedIn

Your notes

Private to this device. Write what you want to remember from today.