Research map / Image generation / Autoregressive and latent generation
Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Vocos, a GAN vocoder that predicts complex STFT coefficients with an isotropic ConvNeXt backbone and inverse STFT upsampling (no transposed convolutions), matching time-domain GAN quality at much higher speed.
Problem
Time-domain GAN vocoders use costly transposed-convolution upsampling and neglect time-frequency inductive bias, while direct generation of complex Fourier coefficients has historically suffered from phase recovery and training instability.
Approach
Generate STFT magnitude and phase directly with a ConvNeXt backbone that keeps a constant temporal resolution, parameterize phase via cos/sin so it implicitly wraps, and upsample only via inverse STFT, trained with mel, feature-matching and hinge adversarial losses.
Key results
- Vocos reaches PESQ 3.70 and ViSQOL 4.66 on LibriTTS, the highest among HiFi-GAN, iSTFTNet and BigVGAN, with UTMOS 3.734 slightly below BigVGAN's 3.749.
- Vocos runs at 6696.52 xRT on GPU versus 98.61 for BigVGAN and 495.54 for HiFi-GAN, about 13x faster than HiFi-GAN and about 70x faster than BigVGAN per the text.
- In subjective tests, Vocos MOS 3.62 and SMOS 4.55 are not significantly different from BigVGAN (MOS 3.64, SMOS 4.54) by Wilcoxon test.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Vocos instead of BigVGAN | neural vocoding from LibriTTS mel-spectrograms · LibriTTS | VISQOL: 4.65 -> 4.66 | Helped |
| use Vocos instead of BigVGAN | neural vocoding from LibriTTS mel-spectrograms · LibriTTS | UTMOS: 3.749 -> 3.734 | Hurt |
| use Vocos instead of BigVGAN | neural vocoding from LibriTTS mel-spectrograms · LibriTTS | MOS: 3.64±{0.15} -> 3.62±{0.15} | No clear effect |
| use Vocos instead of BigVGAN | neural vocoding from LibriTTS mel-spectrograms · LibriTTS | SMOS: 4.54±{0.14} -> 4.55±{0.15} | No clear effect |
| use Vocos instead of BigVGAN | audio reconstruction on music audio · MUSDB18 average | VISQOL: 4.54 -> 4.55 | Helped |
| use Vocos instead of EnCodec | audio reconstruction at 1.5 kbps · DAPS | UTMOS: 1.527 -> 3.210 | Helped |
| use Vocos instead of EnCodec | audio reconstruction at 12 kbps · DAPS | PESQ: 3.283 -> 2.874 | Hurt |
| use Vocos instead of EnCodec | audio reconstruction across bandwidths · DAPS | MOS: 1.09±{0.05} -> 2.73±{0.20}; 1.71±{0.21} -> 3.50±{0.18}; 2.41±{0.15} -> 3.84±{0.16}; 3.08±{0.19} -> 4.00±{0.16} | Helped |
| use Vocos instead of HiFi-GAN | vocoder inference on GPU · Inference | xRT: 495.54 -> 6696.52 | Helped |
| use Vocos instead of HiFi-GAN | vocoder inference on CPU · Inference | xRT: 5.84 -> 169.63 | Helped |
| add implicit phase wrapping instead of predicting absolute phase with scaled tanh (ablation) | Vocos · LibriTTS | UTMOS: 3.590 -> 3.734 | Helped |
| remove Snake activation (ablation) | Vocos · LibriTTS | PESQ: 3.629 -> 3.70 | Helped |
Surprising findings
- Replacing the tanh-bounded absolute phase output with the cos/sin unit-circle parameterization substantially improves PESQ (3.565 to 3.70), showing the periodic inductive bias matters.
- Snake activation, which helped BigVGAN, gave a slight decline for Vocos.
Limitations
- UTMOS evaluation is limited to 16 kHz, and BigVGAN slightly outperforms Vocos on UTMOS.
- MDCT variant performed substantially worse than the STFT-based design.
- Subjective evaluation used only 39 participants with 1560 ratings.
Experimental setup
Datasets: LibriTTS DNS Challenge MUSDB18 DAPS
Benchmarks: LibriTTS MUSDB18 DAPS
Metrics: UTMOS ViSQOL PESQ V/UV F1 periodicity error MOS SMOS xRT
Baselines: HiFi-GAN iSTFTNet BigVGAN EnCodec
Builds on: HiFi-GAN ConvNeXt multi-period discriminator multi-resolution discriminator EnCodec
Closest papers by meaning
- Adversarial Generation of Time-Frequency Features with application in audio synthesis (ICML 2019 · similarity 0.90)
- DegVoC: Revisiting Neural Vocoder from a Degradation Perspective (AAAI 2026 · similarity 0.89)
- Avocodo: Generative Adversarial Network for Artifact-Free Vocoder (AAAI 2023 · similarity 0.88)
- FlowDec: A flow-based full-band general audio codec with high perceptual quality (ICLR 2025 · similarity 0.85)
- Representing Sounds as Neural Amplitude Fields: A Benchmark of Coordinate-MLPs and a Fourier Kolmogorov-Arnold Framework (AAAI 2025 · similarity 0.85)
- Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.