Research map / Large language models / Agents, code and math
Text2midi: Generating Symbolic Music from Captions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces text2midi, an end-to-end model that generates MIDI files from text captions using a frozen FLAN-T5 encoder and an autoregressive transformer decoder over REMI+ tokens, pretrained on pseudo-captioned SymphonyNet and fine-tuned on MidiCaps.
Problem
Generating symbolic MIDI music directly from free-form text descriptions, which prior work handled only via two-stage pipelines or audio generation.
Approach
A single end-to-end encoder-decoder: frozen pretrained FLAN-T5 encodes the caption and a transformer decoder with cross-attention autoregressively generates REMI+ MIDI tokens, with two-stage training using placeholder pseudo-captions on SymphonyNet followed by fine-tuning on MidiCaps.
Key results
- Compression ratio 2.31 vs MuseCoco 2.12 (p<0.0001); MidiCaps ground truth 3.43.
- CLAP score 0.22 vs MuseCoco 0.21 and ground truth 0.26.
- Correct Key with duplicates 35.60% vs MuseCoco 14.59%; tempo bin with tolerance 65.80% vs 54.63%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | compression ratio: 2.12 -> 2.31 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | CLAP score: 0.21 -> 0.22 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | tempo bin (TB): 21.71 -> 39.70 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | tempo bin with tolerance (TBT): 54.63 -> 65.80 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | correct key (CK): 13.70 -> 33.60 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · MidiCaps test set | correct key with duplicates (CKD): 14.59 -> 35.60 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | musical quality: 4.40 -> 4.62 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | overall matching: 4.07 -> 4.67 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | genre matching: 4.40 -> 4.98 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | mood matching: 4.32 -> 5.00 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | key matching: 3.36 -> 3.64 | Helped |
| Use text2midi instead of MuseCoco | Text-to-MIDI generation · Listening study | chord matching: 2.00 -> 2.50 | Helped |
Surprising findings
- Listening test ratings for text2midi were lower than ground truth MidiCaps on all criteria, including chord matching (2.50 vs 3.20).
- The model ignored some instructions such as 'church organ' in the caption, attributed to partial-song training.
Limitations
- Instrumentation control is weak; some requested instruments are ignored.
- Training uses only the first 2,048 tokens (~30-40 s) of each file, limiting data use.
- Data quality of MidiCaps/Lakh MIDI limits output quality.
Experimental setup
Datasets: SymphonyNet MidiCaps
Benchmarks: MidiCaps test subset (100 samples)
Metrics: compression ratio CLAP score tempo bin tempo bin with tolerance correct key correct key with duplicates listening test Likert ratings
Baselines: MuseCoco MidiCaps ground truth
Builds on: FLAN-T5 REMI MidiTok flash attention
Closest papers by meaning
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.89)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.89)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.88)
- SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation (ICML 2025 · similarity 0.87)
- CSL-L2M: Controllable Song-Level Lyric-to-Melody Generation Based on Conditional Transformer with Fine-Grained Lyric and Musical Controls (AAAI 2025 · similarity 0.87)
- Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.