Research map / Vision-language and multimodal / Human and embodied modeling

Toward Multimodal Sheet Music Recognition and Understanding

NeurIPS 2026 · Vision-language and multimodal › Human and embodied modeling · approach: Layout-aware document understanding

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Legato 2 is a system-by-system autoregressive optical music recognition pipeline with YOLO system segmentation, a text-aware byte-fallback ABC tokenizer, and it shows that OMR transcriptions improve frontier VLMs on sheet music understanding.

Problem

Optical music recognition of full-page sheet music scales poorly to long documents and ignores sequential structure, and existing neural OMR cannot transcribe embedded text such as titles and annotations; frontier VLMs also struggle with sheet music understanding.

Approach

Segment pages into systems with YOLO, then autoregressively transcribe each system into a system-level ABC representation conditioned on preceding system outputs (left-truncated context), using a byte-fallback BPE tokenizer that retains embedded text; the resulting transcription is also supplied as context to frontier VLMs.

Key results

• Legato 2 reduces page-level OMR-NED on Camera OpenScore String Quartets to 31.6 vs 58.2 for Legato 1 and 75.4 for Audiveris.
• On text recognition (Camera OpenScore String Quartets), Legato 2 reaches total CER 24.8% vs 58.1% for Gemini and 73.8% for Audiveris.
• Adding Legato 2 transcription as context raises Gemini 3.1 Pro SSMR-Bench accuracy from 71.4 to 92.7 and GPT-5 MusiXQA-style gains from 51.8 to 71.7 on SSMR-Bench.

Surprising findings

• Naive BPE tokenizer trained on text-inclusive data performs far worse than Legato 1 (e.g., 92.1 vs 58.2 OMR-NED on Camera String Quartets).
• Scaling the decoder (deep or wide) or using Qwen-3-VL-2B encoder yields little benefit on real-world datasets, attributed to a distribution shift from synthetic training data.

Limitations

• Trained only on synthetic PDMX-Synth renderings; distribution gap to real-world sheet music limits scaling benefits.
• Lyrics are excluded from transcription; merge step can discard trailing systems on ill-formed output.
• Multi-page evaluation restricted to Legato 1 vs 2 comparison due to computational cost; gains rely on YOLO segmentation quality.

Experimental setup

Datasets: PDMX-Synth PDMX IMSLP OpenScore String Quartets OpenScore Lieder MusiXQA SSMR-Bench

Benchmarks: PDMX-Synth Test Set Rendered OpenScore String Quartets Camera OpenScore String Quartets Rendered OpenScore Lieder Camera OpenScore Lieder IMSLP Piano Scores MusiXQA OMR split SSMR-Bench

Metrics: OMR-NED CER Unordered-CER G-Acc accuracy

Baselines: Legato 1 Audiveris Gemini 3.1 Pro GPT-5 PaddleOCR

Builds on: Legato YOLOv8 Llama-3.2-11B-Vision PDMX-Synth SMT-ABC Notation

Closest papers by meaning

• Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training (NeurIPS 2026 · similarity 0.92)
• Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.87)
• N-gram Unsupervised Compoundation and Feature Injection for Better Symbolic Music Understanding (AAAI 2024 · similarity 0.86)
• Rubato: Transcribing Piano Music with Timestamps (NeurIPS 2026 · similarity 0.86)
• Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.85)
• MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.85)