Research map / Vision-language and multimodal / Cross-modal learning
The Sound of Motions
Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace single frame appearance-only input with multi-frame RGB motion (ablation) | Ours · Instrument mixtures N=2 | SDR: 7.04 -> 7.67 | Improved |
| add optical flow motion feature (ablation) | Ours · Instrument mixtures N=2 | SDR: 7.67 -> 8.05 | Improved |
| replace optical flow with deep dense trajectories via tracking (ablation) | Ours · Instrument mixtures N=2 | SDR: 8.05 -> 8.31 | Improved |
| use motion trajectories instead of appearance only (Sound of Pixels) | Ours vs Sound of Pixels · Instrument mixtures N=3 | SDR: 3.65 -> 4.87 | Improved |
| use motion trajectories instead of appearance only (Sound of Pixels) | Ours vs Sound of Pixels · Instrument mixtures N=4 | SDR: 1.21 -> 3.05 | Improved |
| add curriculum stage 1 (different instrument mixtures) before same-instrument training (ablation) | Ours · Same-instrument duets | SDR: 1.91 -> 3.14 | Improved |
| add curriculum stage 2 (same-kind instruments) (ablation) | Ours · Same-instrument duets | SDR: 3.14 -> 5.72 | Improved |
| add curriculum stage 3 (ablation) | Ours · Same-instrument duets | SDR: 5.72 -> 5.93 | Improved |
| use motion model instead of Sound of Pixels on same-instrument duets | Ours · Same-instrument duets, violin | SDR: 1.95 -> 6.33 | Improved |
| use motion model instead of Sound of Pixels, human evaluation | Ours · Same-instrument duets, erhu | human preference: 44.59% -> 55.41% | Improved |
| use motion model instead of Sound of Pixels, human evaluation | Ours · Same-instrument duets, congas | human preference: 35.42% -> 64.58% | Improved |
Closest papers by meaning
- Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation (NeurIPS 2022 · similarity 0.90)
- Mix and Localize: Localizing Sound Sources in Mixtures (CVPR 2022 · similarity 0.89)
- Music Gesture for Visual Sound Separation (CVPR 2020 · similarity 0.88)
- The Sound of Pixels (ECCV 2018 · similarity 0.87)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.87)
- Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation (ICCV 2023 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
