Research map / Image generation / Autoregressive and latent generation
DiffAR: Denoising Diffusion Autoregressive Model for Raw Speech Waveform Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DiffAR, an autoregressive denoising diffusion model that generates raw speech waveforms frame by frame, conditioned on the previous frame's overlap and optionally on phonemes, durations, energy and pitch.
Problem
Diffusion-based speech synthesis mostly generates spectrograms requiring a vocoder, and existing waveform diffusion models (e.g., DiffWave) produce only fixed-length clips, while end-to-end TTS models lag in quality or are memory-heavy for long text.
Approach
An autoregressive diffusion model over overlapping waveform frames, where each new frame is denoised conditioned on the overlapping samples of the previous frame, allowing arbitrary-duration raw waveform generation with DiffWave-style backbone and optional phoneme/energy/pitch conditioning.
Key results
- DiffAR (1000 steps) reaches MOS 3.77 vs 3.61 for WaveGrad 2 and CER 1.95% vs 3.47%, with 4.65% WER vs 5.75%.
- Against FastSpeech 2, DiffAR (1000 steps) gets MOS 3.82 vs 3.54 and MUSHRA 66.2 vs 63.4, while FastSpeech 2 improved has lower CER/WER (1.73%/4.31%).
- DiffAR has a real-time factor of 31.06, far slower than the baselines (e.g., FastSpeech2 2.79, VITS 0.09), which the authors acknowledge as a limitation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DiffAR (1000 steps) instead of WaveGrad 2 | LJ-Speech TTS · LJ-Speech test | MOS: 3.61\pm 0.09 -> 3.77\pm 0.08 | Helped |
| Use DiffAR (1000 steps) instead of WaveGrad 2 | LJ-Speech TTS · LJ-Speech test | MUSHRA: 63.8\pm 2.3 -> 66.7\pm2.2 | Helped |
| Use DiffAR (1000 steps) instead of WaveGrad 2 | LJ-Speech TTS · LJ-Speech test | CER (%): 3.47 -> 1.95 | Helped |
| Use DiffAR (1000 steps) instead of WaveGrad 2 | LJ-Speech TTS · LJ-Speech test | WER (%): 5.75 -> 4.65 | Hurt |
| Use DiffAR (1000 steps) instead of FastSpeech 2 improved | LJ-Speech TTS · LJ-Speech test | MUSHRA: 64.2\pm 2.5 -> 66.2\pm \textbf{1.5} | Helped |
| Use DiffAR (1000 steps) instead of FastSpeech 2 improved | LJ-Speech TTS · LJ-Speech test | CER (%): \textbf{1.73} -> 1.95 | Hurt |
| Use DiffAR (1000 steps) instead of FastSpeech 2 improved | LJ-Speech TTS · LJ-Speech test | WER (%): \textbf{4.31} -> 4.65 | Hurt |
| Add energy conditioning (ablation) | DiffAR-E (200) · LJ-Speech test with true phonemes and durations | CER (%): 2.90 -> 1.18 | Helped |
| Add pitch conditioning (ablation) | DiffAR (200) · LJ-Speech test with true phonemes, durations, and energy | CER (%): 1.18 -> 1.12 | Helped |
| Increase diffusion steps from 200 to 1000 (ablation) | DiffAR · LJ-Speech test with true phonemes, durations, and energy | CER (%): 1.18 -> 1.70 | Hurt |
| Predict durations instead of using true durations (ablation) | DiffAR-E (200) · LJ-Speech test with true phonemes | CER (%): 2.90 -> 2.68 | Helped |
| Predict phonemes as well as durations (ablation) | DiffAR-E (200) · LJ-Speech test | CER (%): 2.68 -> 3.35 | Hurt |
Surprising findings
- Baseline models such as FastSpeech 2, WaveGrad 2 and several diffusion acoustic models show little or no stochasticity in their outputs, while DiffAR produces slightly different waveforms per inference.
- DiffAR is the only evaluated model that generates vocal fry, which the authors attribute to direct waveform modeling.
Limitations
- Slow synthesis due to sequential autoregressive diffusion; not real-time (RTF 31.06).
- Lower intelligibility than FastSpeech 2 improved in CER/WER; relies on forced aligner, G2P and duration/energy predictors whose errors degrade results.
- Evaluated mainly on single-speaker LJ-Speech; multi-speaker and in-context learning are only briefly explored, and some baselines were unofficial re-implementations.
Experimental setup
Datasets: LJ-Speech VCTK
Benchmarks: LJ-Speech
Metrics: MOS MUSHRA CER WER RTF
Baselines: WaveGrad 2 FastSpeech 2 VITS Grad-TTS ProDiff DiffGAN-TTS
Builds on: DDPM DiffWave FastSpeech 2 WaveGrad 2
Closest papers by meaning
- DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.91)
- Cauchy Diffusion: A Heavy-tailed Denoising Diffusion Probabilistic Model for Speech Synthesis (AAAI 2025 · similarity 0.90)
- DDDM-VC: Decoupled Denoising Diffusion Models with Disentangled Representation and Prior Mixup for Verified Robust Voice Conversion (AAAI 2024 · similarity 0.88)
- Shallow Diffusion for Fast Speech Enhancement (Student Abstract) (AAAI 2024 · similarity 0.88)
- NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Models (ICML 2024 · similarity 0.88)
- Diffusion-TS: Interpretable Diffusion for General Time Series Generation (ICLR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.