Research map / Architectures and efficiency / Attention and transformers
N-gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes NG-Midiformer, which tokenizes MIDI with BPE-based unsupervised compoundation (UCW) and adds an N-gram Transformer encoder whose frequency-weighted outputs are injected into a MIDI-BERT-style encoder, pre-trained with MLM on ~25k pieces.
Problem
Symbolic music understanding with Transformers: REMI yields long sequences that weaken token dependencies, while CP and OctupleMIDI shorten sequences but add [PAD] filler and break short-range dependencies between adjacent events.
Approach
Merge frequent adjacent same-family music events with BPE into variable-length UCW tokens (no [PAD]), and inject frequency-weighted N-gram encoder representations into each layer of the primary Transformer via an N-gram position matrix.
Key results
- NG-Midiformer with UCW-7 reaches 90.63% composer, 81.25% emotion, 65.87% genre, and 63.19% dance accuracy, the best in those tasks in the paper's table.
- On velocity prediction, UCW-4 reaches 54.13% versus 51.63% for MIDI-Bert with CP-4 (reported as state of the art), but melody classification is lower (92.31% vs 96.37% for CP-4 MIDI-Bert).
- Removing N-gram pre-training drops UCW-7 composer accuracy from 90.63% to 58.73%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add N-gram Transformer encoder (NG-Midiformer) on UCW-7 tokens vs MIDI-Bert UCW-7 (ablation) | MIDI-Bert backbone · Pianist8 / EMOPIA / GTZAN / Nottingham | accuracy (%): +14.33, +9.08, +16.31, +15.79 (UCW-7, composer/emotion/genre/dance) | Helped |
| Remove N-gram pre-training stage from NG-Midiformer (ablation) | NG-Midiformer UCW-7 · Pianist8 / EMOPIA / GTZAN / Nottingham | accuracy (%): drops of 31.90, 12.28, 34.84, 10.56 points | Hurt |
| Unsupervised compoundation (UCW-4 tokens) replacing CP-4 under MIDI-Bert (ablation) | MIDI-Bert · Pianist8 / EMOPIA / GTZAN / Nottingham / POP909 velocity and melody | accuracy (%): composer 78.57->66.61, emotion 67.89->63.75, genre 50.49->40.34, velocity 51.63->54.87 (improves), melody 96.37->92.26 | Hurt |
| Use UCW-7 instead of UCW-4 tokens in NG-Midiformer (ablation) | NG-Midiformer · Pianist8 / EMOPIA / GTZAN / Nottingham | accuracy (%): emotion 73.44->81.25, genre 56.67->65.87, dance 55.05->63.19; composer equal at 90.63 | Helped |
| Add N-gram encoder to CP-4 tokens (MIDI-Bert vs NG-Midiformer on CP-4) (ablation) | CP-4 tokens · Pianist8 / EMOPIA / GTZAN / Nottingham / POP909 | accuracy (%): composer 78.57->83.33, genre 50.49->53.89, dance 57.02->61.30; emotion 67.89->65.87 (drop), velocity 51.63->59.82, melod | Mixed |
| Replace CP-7 with UCW-7 in NG-Midiformer on melody/velocity tasks | NG-Midiformer · POP909 melody | accuracy (%): melody 92.31 (UCW-4) vs 96.92 (CP-4 in NG-Midiformer) | Hurt |
Surprising findings
- Without N-gram pre-training, the N-gram model falls below the MIDI-Bert baseline on several tasks, so pre-training is essential to the gains.
- The method only uses the first 512 UCW tokens (about 60% of the CP data) during fine-tuning and still outperforms on most tasks, while underperforming on melody classification.
Limitations
- Only the first 512 UCW tokens are used in fine-tuning, covering about 60% of the CP data, which hurts melody classification.
- Results are reported on small, mostly single-split datasets with a 90/10 train/validation split and no test-set details, and the main table mixes replicated and reported numbers.
- The paper concedes the UCW sequence is longer than CP, so efficiency benefits are modest.
Experimental setup
Datasets: Pop1K7 ADL Piano GiantMIDI-Piano Maestro POP909 Nottingham Pianist8 EMOPIA GTZAN
Benchmarks: Pianist8 EMOPIA GTZAN Nottingham POP909
Metrics: accuracy precision recall
Baselines: MIDI-Bert RNN OM-MIDI-Bert RoAR MidiGPT
Builds on: MIDI-Bert Compound Word REMI ZEN Byte Pair Encoding
Closest papers by meaning
- Toward Multimodal Sheet Music Recognition and Understanding (NeurIPS 2026 · similarity 0.86)
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.86)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.86)
- From Isolated feature to Orbits:\\Discovering Music Concepts via Multi-SAE Alignment (NeurIPS 2026 · similarity 0.85)
- Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.85)
- Text2midi: Generating Symbolic Music from Captions (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.