Research map / Image generation / Autoregressive and latent generation
Compose with Me: Collaborative Music Inpainter for Symbolic Music Infilling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CMI, a human-in-the-loop symbolic music infilling model combining a JEPA-style predictive representation with autoregressive generation (JEP-AGA) and a contrastive Dynamic Interaction Learner that adapts to user-marked unsatisfactory bars.
Problem
Symbolic music infilling that lets users iteratively refine specific bars without regenerating the whole piece, while existing autoregressive infilling models ignore continuous user interaction.
Approach
Augments an autoregressive RWKV-based generator with a pre-trained JEPA-style context encoder and predictor that predict high-level representations of the masked segment, and fine-tunes the encoder-predictor online from user feedback using iteration-wise and sample-wise contrastive losses (DIL), without extra user input.
Key results
- On Bread, CMI reaches PDp 0.83 and CDf 0.88 versus 0.75 and 0.82 for RWKV, with statistically significant gains across all six objective metrics.
- In a 41-listener subjective study, CMI scores 3.73 musicality and 0.49 deception rate versus 3.27 and 0.34 for RWKV.
- Using both iteration-wise and sample-wise DIL yields +20.1% IUS and -43.7% RIC on Bread.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CMI with JEP-AGA | CMI symbolic music infilling · Bread and LMD | pitch, duration, and chord distribution overlap: Bread: 0.74 -> 0.83 (PDp); 0.80 -> 0.88 (CDf). LMD: 0.73 -> 0.83 (PDp); 0.78 -> 0.88 (CDf). | Helped |
| Use CMI instead of RWKV | symbolic music infilling · Bread and LMD | distribution overlap: Bread CDf: 0.82 -> 0.88; LMD CDf: 0.78 -> 0.88 | Helped |
| Add JEP-AGA (ablation) | CMI · music infilling | objective distribution overlap: PDp: 0.74 -> 0.83; CDf: 0.80 -> 0.88 | Helped |
| Add JEP-AGA (ablation) | CMI · music infilling | Musicality: 3.27 -> 3.68 | Helped |
| Add DIL (ablation) | CMI · music infilling | Musicality: 3.68 -> 3.73 | Helped |
| Add DIL (ablation) | CMI · music infilling | Fitness: 3.15 -> 3.10 | Hurt |
| Add DIL (ablation) | CMI · Bread and LMD HITL interaction | IUS: Bread +20.1%; LMD +18.5% | Helped |
| Add DIL (ablation) | CMI · Bread and LMD HITL interaction | RIC: Bread -43.7%; LMD -44.1% | Helped |
| Use both iteration-wise and sample-wise DIL rather than iteration-wise DIL only (ablation) | CMI · Bread | IUS: +6.5% -> +20.1% | Helped |
| Use both iteration-wise and sample-wise DIL rather than sample-wise DIL only (ablation) | CMI · Bread | IUS: +12.2% -> +20.1% | Helped |
| Use both iteration-wise and sample-wise DIL rather than iteration-wise DIL only (ablation) | CMI · LMD | IUS: +13.2% -> +18.5% | Helped |
| Use both iteration-wise and sample-wise DIL rather than sample-wise DIL only (ablation) | CMI · LMD | IUS: +11.7% -> +18.5% | Helped |
Surprising findings
- DIL lowered the fitness subjective score (3.15 to 3.10 in Table 5 comparison context), which the authors attribute to increased creativity at the cost of consistency.
Limitations
- Only autoregressive-based models are studied; diffusion and VAE approaches are not explored.
- Subjective evaluation uses 41 listeners recruited via social media, and user interaction is simulated rather than tested with real users.
- No user interface has been developed yet.
Experimental setup
Datasets: Bread LMD Pop909
Benchmarks: Bread LMD Pop909
Metrics: pitch distribution overlap duration distribution overlap chord distribution overlap deception rate musicality naturalness fitness creativity increase in user satisfaction reduction in interaction cost
Baselines: XLNet-based infilling model Transformer-based infilling model (Malandro) RWKV-based infilling model CMI without DIL
Builds on: JEPA ILM RWKV I-JEPA
Closest papers by meaning
- JEN-1 Composer: A Unified Framework for High-Fidelity Multi-Track Music Generation (AAAI 2025 · similarity 0.88)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.88)
- MusicRL: Aligning Music Generation to Human Preferences (ICML 2024 · similarity 0.87)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.87)
- MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Models (CVPR 2024 · similarity 0.87)
- Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.