AI Paper Atlas

Research map / Video / Video generation

Foley Music: Learning to Generate Music from Videos

Chuang Gan, Deng Huang, Peihao Chen, Joshua B. Tenenbaum, Antonio Torralba

ECCV 2020 · Video › Video generation · approach: Video-to-audio synthesis · 140 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use skeleton keypoints instead of RGB image features (ablation)Foley Music · URMP (violin)NLL of MIDI events: 1.586 -> 1.558Improved
use skeleton keypoints instead of RGB image features (ablation)Foley Music · URMP (trumpet)NLL of MIDI events: 2.748 -> 2.512Improved
use skeleton keypoints instead of optical flow (ablation)Foley Music · URMP (trumpet)NLL of MIDI events: 3.013 -> 2.512Improved
use skeleton keypoints instead of optical flow (ablation)Foley Music · URMP (viola)NLL of MIDI events: 3.859 -> 3.603Improved
use music transformer instead of GRU (ablation)Foley Music · URMP (trumpet)NLL of MIDI events: 2.631 -> 2.512Improved
use music transformer instead of GRU (ablation)Foley Music · URMP (violin)NLL of MIDI events: 1.631 -> 1.558Improved
add hand keypoints to body keypoints (ablation)Foley Music · URMP (trumpet)NLL of MIDI events: 2.805 -> 2.512Improved
add hand keypoints to body keypoints (ablation)Foley Music · URMP (flute)NLL of MIDI events: 2.259 -> 1.995Improved
add hand keypoints to body keypoints (ablation)Foley Music · URMP (violin)NLL of MIDI events: 1.565 -> 1.558Improved
transformer without hand keypoints versus GRU (ablation)Foley Music · URMP (trumpet)NLL of MIDI events: 2.631 -> 2.805Hurt
MIDI-based generation instead of GAN, SampleRNN, WaveNetFoley Music · MUSIC/AtinPiano human real-vs-fakesuccess rate: 12% -> 38%Improved
MIDI-based generation instead of GAN, SampleRNN, WaveNetFoley Music · MUSIC/AtinPiano generated soundNDB: 25 -> 20Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy