Research map / Detection and segmentation / CNN features
Detecting Music Performance Errors with Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Polytune, an end-to-end transformer that takes performance and score audio and outputs annotated MIDI-like tokens marking correct, missed, and extra notes, trained on synthetic-error versions of CocoChorales and MAESTRO.
Problem
Music error detection for beginner musicians relies on automatic alignment (DTW) that breaks on small deviations and overlapping notes, and there is little labeled training data for error detection.
Approach
Replace alignment-plus-comparison with an end-to-end encoder-decoder that encodes score audio and performance audio with two AST encoders fused by a joint encoder and decodes label-annotated MIDI-like tokens via a T5 decoder, plus a synthetic error injection pipeline that creates large labeled datasets (CocoChorales-E, MAESTRO-E).
Key results
- Polytune reaches 64.1% average Error Detection F1 across 14 instruments, about 40 points above the re-implemented baseline.
- Per-category average F1 is 95.0% for Correct, 49.2% for Missed, and 48.0% for Extra, versus 37.0%, 7.6%, and 25.9% for the baseline.
- On MAESTRO-E, the baseline has higher recall on extra notes (87.9% vs 76.3%) but much lower precision, so Polytune wins on F1 (72.0% vs 39.9%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · MAESTRO-E, Correct | F1: 43.5% -> 90.1% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · MAESTRO-E, Missed | F1: 6.6% -> 26.8% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · MAESTRO-E, Extra | F1: 39.9% -> 72.0% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · CocoChorales-E, Correct | F1: 36.7% -> 95.4% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · CocoChorales-E, Missed | F1: 7.7% -> 51.3% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · CocoChorales-E, Extra | F1: 23.5% -> 46.8% | Helped |
| Replace the alignment-based baseline with end-to-end Polytune | Music error detection · MAESTRO-E, Extra | Recall: 87.9% -> 76.3% | Hurt |
| Use self-supervised pretraining (ablation) | Polytune · Music error detection | Error Detection F1 | No clear effect |
Surprising findings
- Self-supervised pretraining gave no benefit to Error Detection F1 in this setting, so the authors trained with direct supervision.
- The re-implemented baseline's higher extra-note recall comes from DTW misalignment marking correct notes as extra, showing that alignment errors inflate recall while hurting precision.
Limitations
- Under-performs on missed notes in homophonic/chordal music, attributed to representing chordal textures in spectrograms.
- Only one synthesizer per instrument in the dataset, which may limit generalization to real timbres.
- Vocabulary is based on 12-tone equal temperament, which may limit non-Western music.
Experimental setup
Datasets: CocoChorales MAESTRO CocoChorales-E MAESTRO-E
Benchmarks: MAESTRO-E CocoChorales-E
Metrics: Error Detection F1 precision recall
Baselines: Benetos et al. (score-informed error detection, re-implemented) Wang et al. (re-implemented) Combined baseline Simply Piano Yousician
Builds on: MT3 Audio Spectrogram Transformer T5 DTW
Closest papers by meaning
- LadderSym: A Multimodal Interleaved Transformer for Music Practice Error Detection (ICLR 2026 · similarity 0.91)
- MuPT: A Generative Symbolic Music Pretrained Transformer (ICLR 2025 · similarity 0.84)
- Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training (NeurIPS 2026 · similarity 0.83)
- Rubato: Transcribing Piano Music with Timestamps (NeurIPS 2026 · similarity 0.83)
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.83)
- MIDI-GPT: A Controllable Generative Model for Computer-Assisted Multitrack Music Composition (AAAI 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.