Audio AI gets less hype than text or vision, but it already powers voice UX, content creation, and accessibility. Three lines: ASR, TTS, music generation.
Three Product Lines
ASR: audio → model → text Whisper, FunASR
TTS: text → model → speech OpenAI TTS, ElevenLabs, ChatTTS
Music: prompt → model → song Suno, Udio, Stable AudioSpeech Recognition (ASR)
Whisper (OpenAI, 2022)
De facto open-source ASR standard:
- 99 languages
- strong English WER (< ~5%)
- runs locally
- speech → English translation mode
Sizes:
tiny (39M) real-time / mobile
base (74M) balanced
small (244M) daily driver
medium (769M) higher accuracy
large (1.5B) best qualityReal Deployments
| Scenario | Stack | Outcome |
|---|---|---|
| Meeting notes | Whisper large + LLM summary | 1h meeting → notes in ~5 min |
| Video captions | Whisper + alignment | standard for creators |
| Call center QA | FunASR (Chinese-optimized) | QA + sentiment |
| Live dictation | Whisper streaming / Deepgram | keyboard replacement |
Chinese ASR Picks
English-heavy → Whisper (free, good enough)
Chinese-heavy → FunASR / Paraformer (lower Chinese WER)
Realtime API → Deepgram / AssemblyAI
Privacy → local Whisper medium + GPUText-to-Speech (TTS)
Evolution
WaveNet → Tacotron → VITS → ChatTTS → OpenAI TTS → ElevenLabs2026 Comparison
| Product | Naturalness | Voice clone | Chinese | Latency | Pricing |
|---|---|---|---|---|---|
| OpenAI TTS | ⭐⭐⭐⭐ | no | fair | low | $15/1M chars |
| ElevenLabs | ⭐⭐⭐⭐⭐ | yes (~3s sample) | good | medium | from $5/mo |
| ChatTTS | ⭐⭐⭐⭐ | no | native | low | open source |
| Fish Speech | ⭐⭐⭐⭐ | yes | native | low | open source |
| Azure TTS | ⭐⭐⭐ | limited | yes | low | usage-based |
Cases
- NotebookLM: docs → two-host podcast-style audio
- Audiobooks: clone narrator voice, batch chapters
- Voice support: ASR + LLM + TTS loop
- Accessibility: enhanced screen reader experiences
Music Generation
2024–2025 breakout:
Suno and Udio
Input: "Chinese folk song about the stars, female vocal, acoustic guitar"
Output: full 2–4 minute track (lyrics + vocals + instrumentation)| Suno v4 | Udio | |
|---|---|---|
| Quality | near pro demo | slightly higher fidelity |
| Chinese | good | weaker |
| Length | up to ~4 min | up to ~2 min |
| Control | tags + description | description + reference audio |
| Free tier | ~50 songs/mo | ~10 songs/mo |
Simplified Architecture
text → text encoder → conditioning
random noise → diffusion/transformer → waveform (or MIDI → render)Like image diffusion, but in time. Hard parts: long structure, vocal/instrument balance, lyric–melody alignment.
Where It Fits
✅ demos, ideation, game/video beds, ad jingles (prototype)
❌ not pro mastering replacement
❌ training-data copyright still unsettledCombined Pipelines
Voice assistant:
speech → Whisper → text → LLM → text → TTS → audio
AI podcast:
document → LLM script → dual-voice TTS → episode
(NotebookLM, iFlytek-style products)OpenAI Realtime API
End-to-end spoken dialogue—no ASR→LLM→TTS chain. GPT-4o voice mode uses this (~300ms latency).
Engineering Picks
ASR → local Whisper (privacy) or Deepgram (realtime)
TTS → ElevenLabs (EN), ChatTTS/Fish Speech (ZH)
Music → Suno (fast ideas), Stable Audio (open/control)
Voice UX → Realtime API or custom ASR+LLM+TTSReferences
Summary
Audio AI: ASR is mature (Whisper everywhere), TTS is product-ready, music gen is exploding (Suno for demos). Combined, you can ship full voice products.