Research map / Image generation / Autoregressive and latent generation
JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JEN-1 Composer, a single audio latent diffusion model that models marginal, conditional and joint distributions over multi-track music via per-track timestep vectors, task prefix prompts, curriculum training, and an interactive human-AI co-composition workflow.
Problem
Text-to-music models generate composite mixes and lack precise, track-level control over multi-track composition, unlike the iterative track-wise workflow of professional composers.
Approach
Extends the Jen-1 audio latent diffusion model to multi-track inputs/outputs (channel-concatenated track latents) with independent per-track timestep vectors, so setting timesteps to T (noise), 0 (clean condition) or a shared t selects marginal, conditional or joint generation; trained with a progressive curriculum from single-track to joint tasks and self-bootstrapped teacher-generated conditioni
Key results
- Mixed-track CLAP of 0.39 vs 0.36 for Jen-1 and 0.35 for MusicGen; RPR of 40% for Jen-1 relative to Composer's outputs.
- Zero-shot FAD on Slakh2100 of 4.04, versus 4.30 for StemGen and 6.55 for MSDM; FAD 3.76 on the proprietary dataset.
- Ablation: individual timestep vectors raise mixed CLAP from 0.28 to 0.33; curriculum training raises it to 0.37; interactive inference reaches 0.39.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use JEN-1 Composer instead of Jen-1 | text-to-music generation · Mixed-track text-to-music generation | CLAP (Bass): 0.19 -> 0.21 | Helped |
| Use JEN-1 Composer instead of Jen-1 | text-to-music generation · Mixed-track text-to-music generation | CLAP (Instrument): 0.29 -> 0.29 | No clear effect |
| Use JEN-1 Composer instead of MusicGen | text-to-music generation · Mixed-track text-to-music generation | CLAP (Melody): 0.33 -> 0.36 | Helped |
| Use JEN-1 Composer instead of Jen-1 | text-to-music generation · Mixed-track text-to-music generation | CLAP (Mixed): 0.36 -> 0.39 | Helped |
| Use JEN-1 Composer instead of StemGen | multi-track generation · Slakh2100 | FAD: 4.30 -> 4.04 | Helped |
| Add individual timestep vectors for each track (ablation) | JEN-1 Composer · Ablation study | CLAP (Melody): 0.28 -> 0.32 | Helped |
| Add individual timestep vectors for each track (ablation) | JEN-1 Composer · Ablation study | CLAP (Bass): 0.20 -> 0.19 | Hurt |
| Add progressive curriculum training (ablation) | JEN-1 Composer with individual timestep vectors · Ablation study | CLAP (Melody): 0.32 -> 0.35 | Helped |
| Add progressive curriculum training (ablation) | JEN-1 Composer with individual timestep vectors · Ablation study | CLAP (Drums): 0.18 -> 0.17 | Hurt |
| Add interactive Human-AI co-composition inference (ablation) | JEN-1 Composer with curriculum training · Ablation study | CLAP (Mixed): 0.37 -> 0.39 | Helped |
Limitations
- Training relies on a proprietary, non-public 800-hour dataset, limiting reproducibility.
- Authors acknowledge that achieving professional music standards remains a challenge.
- Human preference evaluation details (number of raters, samples) are not specified.
Experimental setup
Datasets: proprietary 800-hour multi-track studio recording dataset Slakh2100
Benchmarks: Proprietary test split (160 h) Slakh2100
Metrics: CLAP score FAD Relative Preference Ratio (RPR)
Baselines: MusicLM MusicGen Jen-1 StemGen MSDM
Builds on: Jen-1 Latent Diffusion Model EnCodec Classifier-Free Guidance
Closest papers by meaning
- Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling (NeurIPS 2024 · similarity 0.90)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.89)
- Multi-Source Diffusion Models for Simultaneous Music Generation and Separation (ICLR 2024 · similarity 0.88)
- DITTO: Diffusion Inference-Time T-Optimization for Music Generation (ICML 2024 · similarity 0.88)
- Whole-Song Hierarchical Generation of Symbolic Music Using Cascaded Diffusion Models (ICLR 2024 · similarity 0.88)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.