AI Paper Atlas

Research map / Vision-language and multimodal / Cross-modal learning

The Sound of Motions

Hang Zhao, Chuang Gan, Wei-Chiu Ma, Antonio Torralba

ICCV 2019 · Vision-language and multimodal › Cross-modal learning · approach: Audio-visual source localization · 217 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace single frame appearance-only input with multi-frame RGB motion (ablation)Ours · Instrument mixtures N=2SDR: 7.04 -> 7.67Improved
add optical flow motion feature (ablation)Ours · Instrument mixtures N=2SDR: 7.67 -> 8.05Improved
replace optical flow with deep dense trajectories via tracking (ablation)Ours · Instrument mixtures N=2SDR: 8.05 -> 8.31Improved
use motion trajectories instead of appearance only (Sound of Pixels)Ours vs Sound of Pixels · Instrument mixtures N=3SDR: 3.65 -> 4.87Improved
use motion trajectories instead of appearance only (Sound of Pixels)Ours vs Sound of Pixels · Instrument mixtures N=4SDR: 1.21 -> 3.05Improved
add curriculum stage 1 (different instrument mixtures) before same-instrument training (ablation)Ours · Same-instrument duetsSDR: 1.91 -> 3.14Improved
add curriculum stage 2 (same-kind instruments) (ablation)Ours · Same-instrument duetsSDR: 3.14 -> 5.72Improved
add curriculum stage 3 (ablation)Ours · Same-instrument duetsSDR: 5.72 -> 5.93Improved
use motion model instead of Sound of Pixels on same-instrument duetsOurs · Same-instrument duets, violinSDR: 1.95 -> 6.33Improved
use motion model instead of Sound of Pixels, human evaluationOurs · Same-instrument duets, erhuhuman preference: 44.59% -> 55.41%Improved
use motion model instead of Sound of Pixels, human evaluationOurs · Same-instrument duets, congashuman preference: 35.42% -> 64.58%Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy