Research map / Vision-language and multimodal / Cross-modal learning
The Sound of Pixels
Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, Antonio Torralba
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| predict a mask instead of regressing the spectrogram (ablation) | PixelPlayer · MUSIC synthetic mixtures | NSDR: 5.12 -> 8.87 | Improved |
| use log-frequency scale instead of linear scale (binary mask) (ablation) | PixelPlayer binary mask · MUSIC synthetic mixtures | NSDR: 6.94 -> 8.87 | Improved |
| use log-frequency scale instead of linear scale (ratio mask) (ablation) | PixelPlayer ratio mask · MUSIC synthetic mixtures | NSDR: 6.67 -> 8.56 | Improved |
| use log-frequency scale instead of linear scale (binary mask) (ablation) | PixelPlayer binary mask · MUSIC synthetic mixtures | SAR: 11.12 -> 12.28 | Hurt |
| use binary mask instead of ratio mask (ablation) | PixelPlayer, log scale · MUSIC synthetic mixtures | SIR: 13.75 -> 15.02 | Improved |
| use binary mask instead of ratio mask (ablation) | PixelPlayer, log scale · MUSIC synthetic mixtures | SAR: 14.19 -> 12.28 | Hurt |
| use binary mask instead of ratio mask (ablation) | PixelPlayer · MUSIC human study, sound separation | Correct (%): 39.19 -> 59.11 | Improved |
| use binary mask instead of spectral regression (ablation) | PixelPlayer · MUSIC human study, sound separation | Correct (%): 18.23 -> 59.11 | Improved |
| use binary mask instead of ratio mask (ablation) | PixelPlayer · MUSIC human study, visual-sound correspondence | Yes responses (%): 54.68 -> 67.58 | Improved |
| use binary mask instead of NMF baseline | PixelPlayer · MUSIC synthetic mixtures | NSDR: 3.14 -> 8.87 | Improved |
| use NMF instead of ratio mask model in human study | NMF · MUSIC human study, sound separation | Correct (%): 45.70 -> 39.19 | Mixed |
Closest papers by meaning
- SoundNet: Learning Sound Representations from Unlabeled Video (NeurIPS 2016 · similarity 0.88)
- Learning to Separate Object Sounds by Watching Unlabeled Video (ECCV 2018 · similarity 0.88)
- The Sound of Motions (ICCV 2019 · similarity 0.87)
- Into the Wild with AudioScope: Unsupervised Audio-Visual Separation of On-Screen Sounds (ICLR 2021 · similarity 0.87)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.86)
- Images that Sound: Composing Images and Sounds on a Single Canvas (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
