Research map / Image generation / Autoregressive and latent generation
MusicFlow: Cascaded Flow Matching for Text Guided Music Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MusicFlow, a two-stage cascaded flow-matching model that generates HuBERT semantic features from text and then Encodec acoustic features, trained with masked prediction so it supports text-to-music, continuation and infilling.
Problem
Text-to-music generation is costly for autoregressive and many-step diffusion models, and single-stage models struggle to capture the link between detailed text and music; most models also cannot perform infilling or continuation.
Approach
Bridge text and audio with self-supervised HuBERT semantic features, model them via two non-autoregressive conditional flow-matching networks (text to semantics, then text plus semantics to dense Encodec features) trained with span-masked prediction so the same model does in-context infilling and continuation.
Key results
- On MusicCaps, MusicFlow (330M bidirectional FM+FM) reaches FAD 2.82 vs 3.13 for AudioLDM-2 (746M) and 3.40 for MusicGen (1.5B), with CLAP-text 0.56.
- MusicFlow uses about 50% of MusicGen's network forward passes at inference while outperforming it on FAD (3.13 vs 3.40 at comparable NFE).
- For continuation and infilling with 3s context, FAD drops roughly 50% relative to text-only generation (e.g., continuation FAD 1.63, infilling FAD 1.71).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the HuBERT semantic-prediction stage instead of generating music in one stage (ablation) | MusicFlow flow-matching model · MusicCaps text-to-music | FAD: 4.58 -> 3.37 | Helped |
| Increase second-stage size from 123M to 246M (24L, 768d) (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.37 -> 2.82 | Helped |
| Increase second-stage size from 24L to 32L at 768d (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 2.82 -> 3.12 | Hurt |
| Use 24L, 768d rather than 24L, 1024d in the second stage (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.40 -> 2.82 | Helped |
| Use an 8-layer, 768d first stage rather than a 12-layer, 768d first stage (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.09 -> 2.82 | Helped |
| Train the first stage for 400K rather than 300K updates (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 2.82 -> 2.90 | Hurt |
| Train the second stage for 600K rather than 200K updates (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.19 -> 2.82 | Helped |
| Use HuBERT rather than MERT semantic latents (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.43 -> 2.82 | Helped |
| Use Encodec rather than UniAudio acoustic latents (ablation) | MusicFlow two-stage model · MusicCaps text-to-music | FAD: 3.18 -> 2.82 | Helped |
| Use a first-stage language model rather than bidirectional flow matching (ablation) | MusicFlow · MusicCaps text-to-music | FAD: 2.82 -> 2.69 | Helped |
| Use MusicFlow rather than AudioLDM-2 | MusicCaps text-to-music · MusicCaps | FAD: 3.13 -> 2.82 | Helped |
| Use MusicFlow rather than MusicGen | MusicCaps text-to-music · MusicCaps | FAD: 3.40 -> 2.82 | Helped |
Surprising findings
- Predicting HuBERT semantic features with a smaller first-stage model (84M) works best, and larger first-stage models overfit.
- Single-stage flow matching did not improve when scaled from 246M to 431M parameters, and performed much worse than the two-stage cascade.
Limitations
- Trained on proprietary 20K-hour dataset with predominantly stock music and limited world music, so diversity is limited.
- Still a gap to ground-truth in subjective evaluation; FAD falls behind Jen-1 on MusicCaps.
- Infilling and continuation are compared only against the authors' own text-to-music model because no baselines exist.
Experimental setup
Datasets: proprietary music data (20K hours) MusicCaps
Benchmarks: MusicCaps
Metrics: FAD FD KL-div ISc. CLAP-text CLAP-audio CLAP-sim MOS pairwise preference
Baselines: MusicLM MusicGen AudioLDM-2 Noise2Music JEN-1 UniAudio AudioLDM
Builds on: flow matching HuBERT Encodec AudioBox classifier-free guidance
Closest papers by meaning
- S²MILE: Semantic-and-Structure-Aware Music-Driven Lyric Generation (AAAI 2025 · similarity 0.86)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.86)
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.86)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.86)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.86)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.