Research map / Video / Video generation
PianoMotion10M: Dataset and Benchmark for Hand Motion Generation in Piano Performance
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces PianoMotion10M, a 116-hour piano-performance video dataset with 10M MANO-annotated hand poses, plus a diffusion-based baseline that generates two-hand motion from piano audio via a position predictor and position-guided gesture generator.
Problem
Generating realistic, continuous hand motions and fingering guidance from piano audio is hard, and no large public dataset pairs piano music with hand motion annotations.
Approach
A large bird's-eye-view piano video dataset with MANO hand pose annotations (HaMeR + MediaPipe), and a two-stage model that first predicts 3D hand positions from audio and then generates hand gestures with a diffusion model conditioned on audio and predicted positions.
Key results
- Best model (Our-Large, HuBERT, TF decoder) reaches FID 3.281 vs 4.645 for EmoTalk and 4.157 for LivelySpeaker on the validation set.
- Left-hand FGD drops to 0.372 with the large model versus 0.445 for EmoTalk.
- Denoising with 1000 steps gives the best FID (3.281) and smoothness (0.227) in the step ablation.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use the two-stage position predictor and position-guided gesture generator rather than direct audio-to-pose generation | PianoMotion10M hand motion generation · PianoMotion10M validation | FID: 4.645 -> 3.281 | Helped |
| use the two-stage position predictor and position-guided gesture generator rather than MLP-based diffusion | PianoMotion10M hand motion generation · PianoMotion10M validation | FID: 4.157 -> 3.281 | Helped |
| use TF instead of SSM for the large Wav2vec model (ablation) | Our-Large, Wav2vec · PianoMotion10M validation | FID: 3.453 -> 3.376 | Helped |
| use TF instead of SSM for the large HuBERT model (ablation) | Our-Large, HuBERT · PianoMotion10M validation | FID: 3.395 -> 3.281 | Helped |
| use HuBERT instead of Wav2vec with the large TF decoder (ablation) | Our-Large, TF · PianoMotion10M validation | FID: 3.376 -> 3.281 | Helped |
| use 5 denoising steps instead of 1000 (ablation) | Our-Large, HuBERT, TF · PianoMotion10M validation | FGD: 0.362 -> 0.361 | No clear effect |
| use 10 denoising steps instead of 1000 (ablation) | Our-Large, HuBERT, TF · PianoMotion10M validation | FGD: 0.362 -> 0.363 | Hurt |
| use 100 denoising steps instead of 1000 (ablation) | Our-Large, HuBERT, TF · PianoMotion10M validation | FID: 3.281 -> 3.438 | Hurt |
| use 300 denoising steps instead of 1000 (ablation) | Our-Large, HuBERT, TF · PianoMotion10M validation | FGD: 0.362 -> 0.348 | Helped |
Surprising findings
- Transformer decoders outperform state-space model decoders mainly on the large model; HuBERT only slightly beats Wav2Vec 2.0.
Limitations
- Dataset is imbalanced across 14 subjects; performance levels vary by creator.
- Fails on excessively fast, multi-hand, or non-human piano audio due to out-of-distribution inputs.
- Piano key positions not yet aligned for all videos.
Experimental setup
Datasets: PianoMotion10M
Benchmarks: PianoMotion10M validation set
Metrics: FID FGD WGD PD (position distance) Smoothness
Baselines: EmoTalk LivelySpeaker
Builds on: DDPM MANO HaMeR MediaPipe Wav2Vec 2.0 HuBERT
Closest papers by meaning
- Aria-MIDI: A Dataset of Piano MIDI Files for Symbolic Music Modeling (ICLR 2025 · similarity 0.84)
- Rubato: Transcribing Piano Music with Timestamps (NeurIPS 2026 · similarity 0.84)
- Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion (ICML 2024 · similarity 0.82)
- Structured Multi-Track Accompaniment Arrangement via Style Prior Modelling (NeurIPS 2024 · similarity 0.82)
- Unifying Symbolic Music Arrangement: Track-Aware Reconstruction and Structured Tokenization (NeurIPS 2025 · similarity 0.82)
- MID-FiLD: MIDI Dataset for Fine-Level Dynamics (AAAI 2024 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.