Local Whisper Transcription: Install, Transcribe Audio, and Optimize for Speed & Accuracy

<div className="tldr-box"> <h3>Key Takeaways</h3> <ul> <li>Overview OpenAI's Whisper is a stateoftheart speechtotext model that runs entirely locally.</li> <li>It supports 99+ languages, handles noisy audio surprisingly well, and produces accurate transcripts without sending your audio to any cloud service.</li> <li>This makes it ideal for journalists, researchers, content creators, and anyone handling sensitive audio.</li> <li>Why Local Transcription?</li> </ul> </div>

Overview

OpenAI's Whisper is a state-of-the-art speech-to-text model that runs entirely locally. It supports 99+ languages, handles noisy audio surprisingly well, and produces accurate transcripts without sending your audio to any cloud service. This makes it ideal for journalists, researchers, content creators, and anyone handling sensitive audio.

Why Local Transcription?

| Factor | Cloud (Otter.ai, Rev) | Local Whisper | |---|---|---| | Cost | $10–$30/month or $1–$5/hour | Free | | Privacy | Audio uploaded to servers | Stays on your device | | Speed | Depends on server load | Consistent local speed | | Offline | Requires internet | Works fully offline | | Languages | Typically English only | 99+ languages | | Customization | Limited | Full control |

Installation

Prerequisites

• Python 3.10–3.12
• FFmpeg (for audio processing)
• GPU recommended (NVIDIA with CUDA or Apple Silicon)

Install Whisper

# Install via pip
pip install openai-whisper

# Install FFmpeg if not present
# Windows: choco install ffmpeg (or download from ffmpeg.org)
# macOS: brew install ffmpeg
# Linux: sudo apt install ffmpeg

Verify Installation

whisper --help

Your First Transcription

# Basic transcription (English, small model)
whisper audio.mp3

# Specify language and model size
whisper audio.mp3 --model medium --language en

# Output all formats
whisper audio.mp3 --model large-v3 --output_format all

Model Size Comparison

| Model | Parameters | VRAM | Speed (relative) | Accuracy (English) | Accuracy (Other) | |---|---|---|---|---|---| | tiny | 39M | 1 GB | 32x | ~89% | ~72% | | base | 74M | 1 GB | 16x | ~92% | ~76% | | small | 244M | 2 GB | 6x | ~94% | ~82% | | medium | 769M | 5 GB | 2x | ~96% | ~88% | | large-v3 | 1.55B | 10 GB | 1x | ~98% | ~94% |

When to Use Each Model

| Use Case | Recommended Model | |---|---| | Real-time transcription | tiny or base | | YouTube subtitles | small or medium | | Podcast transcription | medium or large-v3 | | Legal/medical transcription | large-v3 | | Multilingual transcription | large-v3 | | Batch processing many files | small (fastest acceptable quality) |

Advanced Usage

Python API

import whisper

model = whisper.load_model("medium")

# Transcribe a file
result = model.transcribe("interview.wav", language="en")

# Print segmented text with timestamps
for segment in result["segments"]:
    start = segment["start"]
    end = segment["end"]
    text = segment["text"]
    print(f"[{start:.2f}s -> {end:.2f}s] {text}")

# Full text
print(result["text"])

Transcribe with Speaker Diarization

Whisper doesn't natively identify speakers, but you can combine it with pyannote.audio:

# Install: pip install pyannote.audio
from pyannote.audio import Pipeline
from pyannote.core import Segment
import whisper

# Diarization pipeline
diarization_pipeline = Pipeline.from_pretrained(
    "pyannote/speaker-diarization-3.1"
)

# Load audio and run diarization
diarization = diarization_pipeline("audio.wav")

# Transcribe with Whisper
whisper_model = whisper.load_model("large-v3")
transcription = whisper_model.transcribe("audio.wav")

# Align speakers with transcribed segments
for segment in transcription["segments"]:
    time_range = Segment(segment["start"], segment["end"])
    speakers = diarization.crop(time_range)
    speaker = "Unknown"
    for speech in speakers.itertracks(yield_label=True):
        speaker = speech[2]
        break
    print(f"[{speaker}] {segment['text']}")

Batch Processing

Process Multiple Files

import whisper
import os

model = whisper.load_model("medium")
audio_dir = "audio_files/"
output_dir = "transcripts/"

os.makedirs(output_dir, exist_ok=True)

for file in os.listdir(audio_dir):
    if file.endswith((".mp3", ".wav", ".m4a")):
        print(f"Processing {file}...")
        result = model.transcribe(os.path.join(audio_dir, file))

        # Save as text
        output_file = os.path.join(output_dir, file + ".txt")
        with open(output_file, "w", encoding="utf-8") as f:
            f.write(result["text"])

        print(f"Saved to {output_file}")

Shell Script (Bash)

for file in audio/*.mp3; do
  whisper "$file" --model small --output_dir transcripts/
done

Output Formats

# Available formats
whisper audio.mp3 --output_format txt    # Plain text
whisper audio.mp3 --output_format vtt    # WebVTT subtitles
whisper audio.mp3 --output_format srt    # SubRip subtitles
whisper audio.mp3 --output_format tsv    # Tab-separated values
whisper audio.mp3 --output_format json   # JSON with timestamps
whisper audio.mp3 --output_format all    # All formats

Performance Tuning

GPU Acceleration

# Ensure Whisper uses your GPU
whisper audio.mp3 --model large-v3 --device cuda

# On Apple Silicon
whisper audio.mp3 --model large-v3 --device mps

Speed vs. Accuracy Tradeoffs

# Fastest
result = model.transcribe("audio.mp3", 
    language="en",
    fp16=True,                 # Use half precision
    beam_size=1,               # Greedy decoding (fast)
    best_of=1,                 # Single sample
    temperature=0.0            # Deterministic
)

# Most accurate
result = model.transcribe("audio.mp3",
    language="en",
    fp16=False,                # Full precision
    beam_size=5,               # Beam search
    best_of=5,                 # Multiple samples
    temperature=0.8,           # Allow variation
    condition_on_previous_text=True
)

Audio Preprocessing

Better input audio = better transcription:

# Resample to 16kHz mono (Whisper's native format)
ffmpeg -i input.mp3 -ar 16000 -ac 1 output.wav

# Normalize volume
ffmpeg -i input.mp3 -af loudnorm=I=-16:TP=-1.5:LRA=11 normalized.wav

# Remove silence
ffmpeg -i input.mp3 -af silenceremove=1:0:-50dB trimmed.wav

Troubleshooting

| Problem | Solution | |---|---| | Out of memory | Use smaller model or add --fp16 True | | Slow transcription | Enable GPU, use smaller model, reduce beam size | | Poor accuracy | Try large-v3, preprocess audio, specify language | | Wrong language detected | Force language with --language en | | Missing words | Lower temperature to 0.0, increase beam size |

FAQ

Can Whisper run on CPU only?

Yes — but it's slow. A 1-hour audio file takes ~45 minutes with large-v3 on CPU, or ~5 minutes on a mid-range GPU. Use tiny or base models for faster CPU inference.

How accurate is Whisper for non-English languages?

Whisper large-v3 achieves 94%+ WER for major languages (Spanish, French, German, Chinese, Japanese) and 85%+ for lower-resource languages. It significantly outperforms cloud alternatives in many languages.

Can I fine-tune Whisper on my domain?

Yes — Whisper supports fine-tuning with Hugging Face Transformers. Fine-tuning on medical or legal terminology can improve accuracy by 5–15% in those domains.