Research map / Image generation / Autoregressive and latent generation

AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency

NeurIPS 2026 · Image generation › Autoregressive and latent generation · approach: Neural text-to-speech synthesis

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes AnyEdit, a separation-free two-stage framework (instruction-guided AR editor on noise-robust prosody tokens plus in-context flow-matching renderer) for speech and singing editing in noisy, reverberant real-world audio, and releases the AnyEditBench benchmark.

Problem

Localized editing of speech and singing recordings in real-world acoustic environments (noise, reverberation, background music) while preserving speaker identity, prosody, and the surrounding acoustic scene, without source separation.

Approach

Decouple semantic editing (AR prediction of clean prosody and content-style tokens, with a teacher-student noise-robust prosody tokenizer) from acoustic rendering (flow-matching transformer conditioned on the source acoustic context), add unmodified-area-aware loss weighting, and align the AR editor with DPO on real-to-real singing pairs mined via Chromagram-DTW from song repetitions.

Key results

• On the AnyEditBench real-world subset, AnyEdit reports English speech WER 0.0616 and SIM-spk 0.9675, and English SIM-bg 0.7705 vs 0.6418 for Step-Audio-EditX.
• At 0 dB SNR, SNR estimation error for speech drops from 13.321 (Vevo2) to 3.368 (AnyEdit).
• In subjective evaluation (10 raters, 60 samples), AnyEdit obtains N-MOS 4.48 and S-MOS 4.45 for speech, and a 46.5% fooling rate.

Surprising findings

• Ablation shows that removing the FMT's noisy-data rendering drops SIM-bg to 0.200 despite near-unchanged WER, and that replacing mined real-to-real pairs with SVS-synthesized targets gives DPO results nearly equivalent to no alignment.

Limitations

• Mined real-to-real pairs require songs with repetitive structure (common in pop music).
• Does not handle modification of high-level expressive performance styles, deferred to future work.
• Several baselines are not applicable on some subsets (e.g., VoiceCraft English only; SoulX-Singer lyric editing had no public code and was reimplemented by the authors).

Experimental setup

Datasets: Emilia SingNet DNS Challenge WHAM! AudioSet LibriTTS VCTK AISHELL-1 YODAS SingVERSE

Benchmarks: AnyEditBench

Metrics: WER CER SIM-spk F0 Corr SIM-bg RT60Error SNRError N-MOS S-MOS Fooling Rate

Baselines: SSR-Speech VoiceCraft Step-Audio-EditX StyleSinger TCSinger SoulX-Singer Ming-UniAudio MiMo-Audio Vevo2

Builds on: VoiceCraft Vevo2 RepCodec Qwen2.5-0.5B-Instruct DPO

Closest papers by meaning

• SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.88)
• NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizers (ICLR 2024 · similarity 0.87)
• SAO-Instruct: Free-form Audio Editing using Natural Language Instructions (NeurIPS 2025 · similarity 0.86)
• SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.86)
• StyleSinger: Style Transfer for Out-of-Domain Singing Voice Synthesis (AAAI 2024 · similarity 0.86)
• Prosody-Enhanced Acoustic Pre-training and Acoustic-Disentangled Prosody Adapting for Movie Dubbing (CVPR 2025 · similarity 0.86)