Research map / Architectures and efficiency / RNNs and normalization

Multi-band Frequency Reconstruction for Neural Psychoacoustic Coding

ICML 2025 · Architectures and efficiency › RNNs and normalization · approach: Sparse dictionary representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes MUFFIN, a fully convolutional neural psychoacoustic codec whose multi-band spectral RVQ quantizes latent frequency bands with a modified snake activation, reaching 12.5 Hz tokens and outperforming HiFi-Codec and others on reconstruction and zero-shot TTS.

Problem

High-fidelity, perceptually preserving neural audio compression across speech, music and general sound, at low token rates, without the inefficiency of full-band RVQ and with usable tokens for speech LLMs.

Approach

Apply FFT-based frequency band splitting inside the encoder's latent space and quantize each band with successive residual codebooks (MBS-RVQ), so each codebook captures a psychoacoustically motivated band; add a bias term to the snake activation and use a HiFi-Codec-style convolutional backbone.

Key results

• On LibriTTS test-clean at 3.0 kB/s, MUFFIN gets PESQ 2.996 and STOI 0.954 versus HiFi-Codec's 2.712 and 0.943.
• The 12.5 Hz MUFFIN variant reaches UTMOS 4.074 on LibriTTS test-clean, above Mimi's 2.966 at a similar token rate.
• In zero-shot VALL-E TTS, MUFFIN tokens give WER 12.20% versus 32.35% for HiFi-Codec and 21.05% for Encodec.

Measured effects reported in the paper

Surprising findings

• Codebook 1 alone carries most speech intelligibility, with WER above 70% when decoding from any single later codebook, unlike vanilla RVQ's distribution.
• Instrumental music does not separate cleanly across codebooks, so the psychoacoustic disentanglement works mainly for vocal content.

Limitations

• Multi-band splitting in the latent space is not theoretically proven to preserve psychoacoustic properties; the Lipschitz argument is informal.
• Instrumental music is poorly disentangled across codebooks, and highly compressed variants lose PESQ/STOI on music.
• The VALL-E experiments use a reimplementation trained on under 600 hours, with a modified prompt protocol, so results are not directly comparable to the original paper.

Experimental setup

Datasets: LibriTTS EARS Music4All ESC-50 VoxCeleb LibriSpeech

Benchmarks: LibriTTS test-clean LibriTTS test-other IEMOCAP GTZAN BBC sound effects LibriSpeech test-clean

Metrics: STFT distance Mel distance PESQ STOI UTMOS ViSQOL F0CORR WER MOS S-MOS

Baselines: OPUS Encodec DAC HiFi-Codec Mimi VALL-E w/ Encodec VALL-E w/ HiFi-Codec

Builds on: HiFi-Codec SoundStream EnCodec VALL-E snake activation

Closest papers by meaning

• Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.85)
• FlowDec: A flow-based full-band general audio codec with high perceptual quality (ICLR 2025 · similarity 0.84)
• FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks (NeurIPS 2025 · similarity 0.84)
• WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio Language Modeling (ICLR 2025 · similarity 0.84)
• Scaling Transformers for Low-Bitrate High-Quality Speech Coding (ICLR 2025 · similarity 0.84)
• Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis (ICLR 2024 · similarity 0.84)