Research map / Image generation / Autoregressive and latent generation
Rubato: Transcribing Piano Music with Timestamps
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Rubato, a prompt-conditioned encoder-decoder (Canary-180M-Flash architecture, trained from scratch) that transcribes piano audio directly into timestamped sheet music using the InterMo text notation and multitask dialects.
Problem
Converting piano recordings into engraved sheet music with timestamps currently relies on staged cascades (AMT then MIDI-to-score), which lose notational structure and temporal grounding; no prior work does time-aligned score transcription end-to-end.
Approach
Joint end-to-end prediction of multi-staff polyphonic notation and timestamps via InterMo, a canonical text serialization of sheet music (intervals and moments) with task-specific dialects enabling multitask training from heterogeneous datasets (MAESTRO, (n)ASAP, PDMX) under prompt tokens.
Key results
- Rubato (TAST) achieves OMR-NED 75.9 on ATEPP and 64.3 on ASAP, lower than every cascade baseline including oracle-MIDI variants (best cascade Tkun→M2ST 85.2 on ATEPP).
- Rubato (DBD) beat F1 of 82.6 vs Beat-This 79.9 on ASAP; Rubato (AMT) note F1 97.0 on MAESTRO vs Tkun 98.3.
- Rubato (TAST) with relative timestamps reaches MAP_performer 72.3 on ATEPP while keeping MAP_work at 96.6.
Surprising findings
- Replacing predicted MIDI with oracle Disklavier MIDI does not change OMR-NED for MIDI cascades, suggesting downstream structure recovery dominates error.
- Gemini 3.1 Pro achieves near-100% OMR-NED (97.6–98.9) on ASAP, largely by recalling memorized scores rather than transcribing audio.
Limitations
- TAST note F1 (91.0 on ASAP, 87.1 on MAESTRO) trails dedicated AMT systems because it transcribes written rather than performed notes.
- Rubato (AMT) KeyOff F1 is 73.0, below Tkun's specialist finetune.
- Training uses synthesized audio from PDMX; test sets are real recordings only.
Experimental setup
Datasets: MAESTRO v3 (n)ASAP PDMX ATEPP ASAP-Beyer
Benchmarks: ASAP ASAP-Beyer ATEPP MAESTRO
Metrics: OMR-NED F1 (beat, downbeat, note) MAP_work MAP_performer
Baselines: Piano-A2S PM2S M2ST Tkun Bytedance Aria-AMT MT3 Beat-This Gemini 3.1 Pro CLEWS CoverHunterC
Builds on: Canary-180M-Flash SentencePiece UnigramLM subword regularization Piano-A2S M2ST
Closest papers by meaning
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.91)
- Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training (NeurIPS 2026 · similarity 0.86)
- Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.86)
- Toward Multimodal Sheet Music Recognition and Understanding (NeurIPS 2026 · similarity 0.86)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.86)
- Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.