Research map / Label-efficient and robust learning / Self-supervised and contrastive

DiffNorm: Self-Supervised Normalization for Non-autoregressive Speech-to-speech Translation

NeurIPS 2024 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Self-supervised speech and brain decoding · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DiffNorm, a latent-diffusion-based self-supervised normalization of target speech units, plus classifier-free guidance regularization for CMLM-based non-autoregressive speech-to-unit translation, improving ASR-BLEU on CVSS En-Es and En-Fr.

Problem

Non-autoregressive speech-to-speech translation suffers from incoherent and repetitive outputs (multi-modality problem) because target speech unit distributions are complex and multi-modal.

Approach

Use a VAE plus latent diffusion model trained with a self-supervised noise-estimation objective (plus reconstruction and NLL terms) to denoise synthetically corrupted target speech features into normalized units for NAT training, and apply classifier-free guidance by randomly dropping source encoder representations during CMLM training and mixing conditional/unconditional distributions at inferenc

Key results

• CMLM + DiffNorm + CG reaches 19.49 ASR-BLEU on En-Es vs 12.58 for CMLM and 13.75 for the Conformer autoregressive model, with 5.34x speedup at 15 iterations.
• On En-Fr, CMLM + DiffNorm + CG reaches 17.54 ASR-BLEU vs 15.62 for CMLM and 17.07 for the Conformer autoregressive model.
• Ablation shows noise injection perturbing about 20% of units (T=100 for En-Es, T=120 for En-Fr) gives the best downstream performance.

Measured effects reported in the paper

Surprising findings

• Classifier-free guidance gives larger gains on original units than on DiffNorm-normalized units, since normalized units already improve quality substantially.
• Setting the CG guidance weight to zero (plain CMLM decoding) still outperforms standard CMLM when the model was trained with CG.

Limitations

• Data pre-processing cost of DiffNorm requires running the diffusion model on all training samples, which may be large for bigger datasets.
• Evaluation limited to two CVSS language pairs (En-Es, En-Fr).
• Broader risk of speaker mimicry from speech synthesis systems.

Experimental setup

Datasets: CVSS-C En-Es CVSS-C En-Fr

Benchmarks: CVSS-C En-Es CVSS-C En-Fr

Metrics: ASR-BLEU accuracy of reconstructed units decoding speed (units/second) speedup

Baselines: Transformer (S2UT) Conformer S2UT Norm Transformer CMLM CMLM + Bilateral Perturbation

Builds on: CMLM DDPM DDIM classifier-free guidance Bilateral Perturbation

Closest papers by meaning

• Speech Self-Supervised Learning Using Diffusion Model Synthetic Data (ICML 2024 · similarity 0.89)
• DiTAR: Diffusion Transformer Autoregressive Modeling for Speech Generation (ICML 2025 · similarity 0.89)
• DELTA-TTS: Adapting Autoregressive Model into a Diffusion Language Model for Text-to-Speech (NeurIPS 2026 · similarity 0.88)
• AEQA-NAT : Adaptive End-to-end Quantization Alignment Training Framework for Non-autoregressive Machine Translation (ICML 2025 · similarity 0.87)
• DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factors (ICLR 2025 · similarity 0.87)
• TaDiCodec: Text-aware Diffusion Speech Tokenizer for Speech Language Modeling (NeurIPS 2025 · similarity 0.87)