Research map / Image generation / Autoregressive and latent generation
Drop the Beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Freestyler, a three-stage system (LM over semantic tokens conditioned on lyrics and accompaniment, conditional flow matching for mel-spectrograms, BigVGAN-V2 vocoder) that generates rap vocals from lyrics and beats with zero-shot timbre from a 3-second reference, plus the RapBank dataset.
Problem
Rap vocal generation is underexplored; existing singing voice synthesis needs notes and durations, and text-to-song generates vocals from lyrics alone, which compromises rhythmic alignment with accompaniment.
Approach
Generate rap vocals conditioned on lyrics and fine-grained accompaniment features (with the accompaniment shifted left by K frames for future rhythmic context and randomly masked during training), using discrete SSL-based semantic tokens as a proxy between lyrics and spectrograms so later stages can be trained without labels.
Key results
- Freestyler achieves MOS-N 3.88 and MOS-M 3.84 with accompaniment, versus 3.23 and 3.06 for randomly mixed ground-truth samples.
- Removing random accompaniment masking raises WER from 33.2 to 56.1 and drops MOS-N to 3.35.
- Using rap reference audio in both LM and CFM stages gives SECS 0.69 and MOS-S 3.86, versus 0.50-0.53 SECS when only one stage is conditioned.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add accompaniment conditioning during training and inference | Freestyler · RapBank evaluation | MOS-N: 3.92 -> 3.43 | Hurt |
| add accompaniment conditioning during training and inference | Freestyler · RapBank evaluation | MOS-R: 3.80 -> 3.21 | Hurt |
| add accompaniment conditioning (ablation) | Freestyler · RapBank vocals mixed with accompaniment | MOS-M: 3.19 -> 3.84 | Helped |
| add supervised finetuning (ablation) | Freestyler · RapBank vocals mixed with accompaniment | MOS-N: 3.80 -> 3.88 | Helped |
| add random masking of the accompaniment condition (ablation) | Freestyler · RapBank vocals mixed with accompaniment | MOS-N: 3.35 -> 3.88 | Helped |
| add random masking of the accompaniment condition (ablation) | Freestyler · RapBank vocals mixed with accompaniment | WER: 56.1 -> 33.2 | Helped |
| add reference audio conditioning in both LM and CFM (ablation) | Freestyler · RapBank, rap reference | MOS-S: 3.44 -> 3.86; 3.37 -> 3.86 | Helped |
| add reference audio conditioning in both LM and CFM (ablation) | Freestyler · RapBank, rap reference | SECS: 0.53 -> 0.69; 0.50 -> 0.69 | Helped |
| use speech instead of rap as the reference audio | Freestyler · RapBank, zero-shot timbre control | MOS-S: 3.86 -> 3.64 | Mixed |
Surprising findings
- Training with accompaniment but inferring without it still improves naturalness and rhythmicity over never using accompaniment (MOS-R 3.55 vs 3.21).
- Speech reference audio achieves high subjective similarity (MOS-S 3.64) but low objective SECS (0.28).
Limitations
- Evaluation limited to English rap; other languages (84 languages in RapBank) are left to future work.
- WER is high for all models including ground truth because Whisper is not trained on singing data, so intelligibility metrics are coarse.
- Model has no duration conditions, so generated vocals do not precisely match ground-truth timing.
Experimental setup
Datasets: RapBank
Benchmarks: RapBank held-out test set (200 samples)
Metrics: MOS-N MOS-M MOS-R MOS-S WER SECS FAD KLD CLAP score
Baselines: GT Rand Mix w/o Acco w/o SFT w/o Mask
Builds on: Matcha-TTS BigVGAN-V2 Wav2Vec XLS-R Whisper MelodyLM
Closest papers by meaning
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.87)
- SongCreator: Lyrics-based Universal Song Generation (NeurIPS 2024 · similarity 0.85)
- GTSinger: A Global Multi-Technique Singing Corpus with Realistic Music Scores for All Singing Tasks (NeurIPS 2024 · similarity 0.85)
- TechSinger: Technique Controllable Multilingual Singing Voice Synthesis via Flow Matching (AAAI 2025 · similarity 0.84)
- SongEditor: Adapting Zero-Shot Song Generation Language Model as a Multi-Task Editor (AAAI 2025 · similarity 0.84)
- LeVo: High-Quality Song Generation with Multi-Preference Alignment (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.