Research map / Image generation / Autoregressive and latent generation
SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SongEditor, a language-model-based song editing paradigm (segment-wise and track-wise editing of lyrics, vocals, and accompaniment) built on a zero-shot song generation base model SongLM.
Problem
Partial editing of existing songs (re-writing segments or replacing vocal/accompaniment tracks) is underexplored; current song generation models synthesize full songs from scratch and cannot edit.
Approach
Adds editing to a Llama-style autoregressive song language model over a two-branch semantic tokenizer (MERT for accompaniment, HuBERT for vocals) with a DiT diffusion decoder, using a context-free lyric infilling strategy with token rearrangement, force-smoothing training, score-based candidate selection, and a gated multi-source encoder for track-wise completion.
Key results
- Segment-wise editing: SongEditor reaches 20.82% PER and 1.30 FAD vs VoiceCraft's 35.23% PER and 1.34 FAD, with higher musicality (3.25 vs 3.03) and coherence (3.82 vs 3.48).
- Ablating force-smoothing raises PER from 20.82% to 25.68% and worsens FAD and MOS.
- Vocal-to-song track-wise editing with context reduces FAD from 2.71 to 1.13 and gives intelligibility MOS 4.33.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add segment-wise editing capability to SongLM | SongLM · Song generation test set | PER (%): 20.63 -> 18.33 | Helped |
| Add segment-wise editing capability to SongLM | SongLM · Song generation test set | FAD: 1.99 -> 2.24 | Hurt |
| Add segment-wise editing capability to SongLM | SongLM · Song generation test set | Musicality: 2.95 -> 3.02 | Helped |
| Use context-free lyrics instead of full contextual lyrics | SongEditor · Segment-wise editing | PER (%): 35.23 -> 20.82 | Helped |
| Use context-free lyrics instead of full contextual lyrics | SongEditor · Segment-wise editing | Coherence: 3.48 -> 3.82 | Helped |
| Use context-free lyrics instead of full contextual lyrics | SongEditor · Segment-wise editing | Smoothness: 3.63 -> 3.53 | Hurt |
| Add force-smoothing training (ablation) | SongEditor · Segment-wise editing | PER (%): 25.68 -> 20.82 | Helped |
| Add force-smoothing training and score-based candidate selection (ablation) | SongEditor · Segment-wise editing | Smoothness: 3.43 -> 3.53 | Helped |
| Add score-based candidate selection (ablation) | SongEditor · Segment-wise editing | Smoothness: 3.50 -> 3.53 | Helped |
| Provide vocal track as input rather than neither source track | SongEditor+ · Track-wise generation without context | Musicality: 2.68 -> 3.38 | Helped |
| Provide accompaniment track as input rather than neither source track | SongEditor+ · Track-wise generation without context | FAD: 2.61 -> 1.29 | Helped |
| Add context to vocal-track input | SongEditor+ · Vocal-to-song generation | FAD: 2.71 -> 1.13 | Helped |
Surprising findings
- Smoothness MOS decreases for SongEditor relative to VoiceCraft (3.53 vs 3.63), which the authors attribute to temporal boundary annotation inaccuracies.
- Accompaniment-to-song generation yields a higher FAD than the vocal condition (1.29 vs 1.13 for vocal-to-song with context), and vocal-only tracks give a markedly lower intelligibility score for accompaniment-conditioned generation.
Limitations
- Regenerates specified sentences without additional control over the editing segment.
- No explicit decoupling of tracks at the semantic level.
- Smoothness decline relative to the VoiceCraft baseline attributed to annotation inaccuracies.
Experimental setup
Datasets: in-house song dataset of ~700K songs (50K hours) BSRNN-separated vocals
Benchmarks: in-house 200-sample test set (90-120 s songs)
Metrics: PER FAD MOS musicality MOS quality MOS coherence MOS smoothness MOS intelligibility
Baselines: VoiceCraft SongLM
Builds on: VoiceCraft MusicLM-style semantic tokens Latent Diffusion Model DiT BS-RNN
Closest papers by meaning
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.90)
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.89)
- SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement (NeurIPS 2025 · similarity 0.88)
- AnyEdit: A Unified Framework for Speech and Singing Voice Editing with Real-World Environmental Consistency (NeurIPS 2026 · similarity 0.88)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.87)
- SongGLM: Lyric-to-Melody Generation with 2D Alignment Encoding and Multi-Task Pre-Training (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.