Research map / Image generation / Fast diffusion sampling
RFWave: Multi-band Rectified Flow for Audio Waveform Reconstruction
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes RFWave, a multi-band rectified flow vocoder that generates complex STFT spectrograms at the frame level with 10 sampling steps, reaching up to 160x real-time on an RTX 4090.
Problem
Diffusion-based audio waveform reconstruction is slow because it needs many sampling steps and operates at the sample-point level, making it much slower than GAN vocoders.
Approach
Combines rectified flow (straight-trajectory ODE) with frame-level multi-band complex spectrogram generation in a single shared ConvNeXtV2 model, plus energy-balanced loss, overlap loss, STFT loss, and an equal-straightness selection of Euler time points.
Key results
- On LibriTTS, RFWave reaches MOS 3.95 vs PriorGrad 3.75 and FreGrad 2.99 on the combined Mel test sets, with PESQ 4.202.
- On out-of-domain MUSDB18 (trained on LibriTTS), RFWave averages MOS 3.67 vs BigVGAN 3.51 and Vocos 3.10.
- RFWave runs at 162.59x real-time on an RTX 4090 with 10 sampling steps and 780 MB GPU memory, versus BigVGAN at 72.68x.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use time-domain rather than frequency-domain modeling (ablation) | RFWave · LJSpeech | PESQ: 3.872 -> 4.127 | Helped |
| add energy-balanced loss (ablation) | RFWave · LJSpeech | PESQ: 4.127 -> 4.181 | Helped |
| add overlap loss (ablation) | RFWave · LJSpeech | PESQ: 4.181 -> 4.158 | Hurt |
| add STFT loss (ablation) | RFWave · LJSpeech | PESQ: 4.158 -> 4.211 | Helped |
| add equal-straightness sampling (ablation) | RFWave · LJSpeech | PESQ: 4.211 -> 4.275 | Helped |
| replace Rectified Flow with DDPM using 50 sampling steps (ablation) | RFWave · LJSpeech | PESQ: 4.211 -> 2.790 | Hurt |
| replace ConvNeXtV2 with ResNet (ablation) | RFWave · LJSpeech | PESQ: 4.211 -> 3.865 | Hurt |
| reduce ConvNeXtV2 backbone size by half (ablation) | RFWave · LJSpeech | GPU xRT: 162.59 -> 228.70 | Helped |
| double ConvNeXtV2 backbone size (ablation) | RFWave · LJSpeech | PESQ: 4.211 -> 4.285 | Helped |
| double ConvNeXtV2 backbone size (ablation) | RFWave · LJSpeech | GPU xRT: 162.59 -> 111.62 | Hurt |
| use classifier-free guidance for EnCodec token conditioning | RFWave · EnCodec tokens | MOS | Helped |
| use classifier-free guidance for Mel-spectrogram conditioning | RFWave · Mel-spectrograms | audio quality | No clear effect |
Surprising findings
- Standard DDPM noise schedule with 50 steps performed worse than the rectified flow baseline with far fewer steps, highlighting rectified flow's importance.
- STFT loss improves PESQ but worsens ViSQOL and periodicity error.
Limitations
- Metrics do not consistently correlate with human judgments, per the authors.
- Quality on EnCodec tokens depends on CFG; ViSQOL remains highest for EnCodec's GAN decoder.
- Speed benchmarks use no specific hardware optimizations and a single RTX 4090.
Experimental setup
Datasets: LibriTTS MTG-Jamendo Opencpop LJSpeech Common Voice 7.0 DNS Challenge 4 FSD50K AudioSet MUSDB18
Benchmarks: LibriTTS MTG-Jamendo Opencpop MUSDB18 LJSpeech Custom universal EnCodec test set (900 samples from 15 datasets)
Metrics: MOS PESQ ViSQOL V/UV F1 Periodicity error xRT speed GPU memory
Baselines: PriorGrad FreGrad Vocos BigVGAN EnCodec MBD
Builds on: Rectified Flow ConvNeXtV2 PriorGrad Multi-Band Diffusion Vocos
Closest papers by meaning
- DFlow: A Generative Model Combining Denoising AutoEncoder and Normalizing Flow for High Fidelity Waveform Generation (ICML 2024 · similarity 0.85)
- BinauralFlow: A Causal and Streamable Approach for High-Quality Binaural Speech Synthesis with Flow Matching Models (ICML 2025 · similarity 0.85)
- PeriodWave: Multi-Period Flow Matching for High-Fidelity Waveform Generation (ICLR 2025 · similarity 0.84)
- MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generation (ICLR 2025 · similarity 0.84)
- Taming Rectified Flow for Inversion and Editing (ICML 2025 · similarity 0.84)
- Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decomposition (CVPR 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.