AI Paper Atlas

Research map / Video / Video segmentation and tracking

Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation

Ziyang Chen, Shengyi Qian, Andrew Owens

ICCV 2023 · Video › Video segmentation and tracking · approach: Sign language and physiological video analysis · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add reflection-invariant (permutation invariant) geometric loss to handle front-back ambiguity (ablation)SLfM · HM3D-SS LibriSpeechcamera angle MAE (deg): 26.80 -> 0.77Improved
add binaural IID loss (ablation)SLfM · HM3D-SS LibriSpeechcamera angle MAE (deg): 29.20 -> 6.99Improved
add symmetry loss on top of geometric and binaural losses (ablation)SLfM · HM3D-SS LibriSpeechcamera angle MAE (deg): 6.99 -> 0.77Improved
pretrain features with cross-view binauralization instead of random features (frozen) (ablation)SLfM · HM3D-SScamera angle MAE (deg): 29.26 -> 0.77Improved
finetune features during SLfM instead of freezing (ablation)SLfM · HM3D-SSaudio angle MAE (deg): 3.17 -> 2.77Improved
use both visual and audio features in binauralization pretext (ablation)cross-view binauralization · HM3D-SS LibriSpeech pretextSTFT distance: 0.206 -> 0.130Improved
train pretext with 3 target views instead of 2 (ablation)cross-view binauralization · HM3D-SS LibriSpeech linear probecamera rotation accuracy (%): 80.0 -> 81.3Improved
predict right channel from left (L2R) instead of mono-to-binaural (ablation)cross-view binauralization · HM3D-SS LibriSpeech linear probeaudio localization accuracy (%): 75.4 -> 78.5 audio; 81.3 -> 76.1 rotationMixed
use cross-attention U-Net instead of 2.5D U-Net for audio prediction (ablation)cross-view binauralization · HM3D-SS LibriSpeechdownstream camera rotation accuracy (%): 80.0 -> 75.3Hurt
increase reverberation (RT60) in rendered audio (ablation)SLfM · HM3D-SSSLfM accuracyHurt
geometric loss weight lambda in range 3 to 8 (ablation)SLfM · HM3D-SSSLfM performanceNo clear effect
use SLfM-trained visual model instead of SIFT/Superglue feature matchingSLfM · Stanford2D3Dmean rotation error (deg): 5.07 -> 1.14 (vs Superglue)Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy