Research map / Vision-language and multimodal / Human and embodied modeling
Cyclic Learning for Binaural Audio Generation and Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLUP, a cyclic framework where an unsupervised object-level sounding-object localizer guides a diffusion-based binaural upmixing model, and the generated binaural audio in turn improves localization.
Problem
Visually guided binaural audio generation from mono audio typically uses the whole scene as guidance, which includes silent objects and background, and the spatial cues in binaural audio are not exploited to improve sounding object localization.
Approach
Couples unsupervised object-level and scene-level sounding object localization (OSA module) with a diffusion waveform upmixing model guided by localized objects, plus a semantic-spatial mining module (SSM) that feeds generated binaural audio back into localization, forming a cyclic learning loop.
Key results
- On FAIR-Play, the method reaches STFT 0.787 vs. 0.851 for the prior best (SAGM), ENV 0.128 vs. 0.134, WAV 5.244 vs. 5.684, and SNR 7.546 vs. 7.044.
- On YT-Music, STFT 0.856 vs. 0.875 for SAGM, with SNR 5.711 vs. 5.601.
- In the localization ablation, using binaural audio with SSM gives the best localization accuracy (97.4%) versus 96.1% for mono audio.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use object-aware cyclic binaural generation instead of SAGM | binaural audio generation · FAIR-Play | STFT: 0.851 -> 0.787 | Helped |
| use object-aware cyclic binaural generation instead of SAGM | binaural audio generation · FAIR-Play | SNR: 7.044 -> 7.546 | Helped |
| use object-aware cyclic binaural generation instead of SAGM | binaural audio generation · YT-Music | STFT: 0.875 -> 0.856 | Helped |
| use object-aware cyclic binaural generation instead of SAGM | binaural audio generation · YT-Music | SNR: 5.601 -> 5.711 | Helped |
| remove visual guidance (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.779 -> 1.078 | Hurt |
| use full visual guidance only (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.779 -> 0.786 | No clear effect |
| use localized sounding objects only (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.779 -> 0.828 | Hurt |
| use scene-level localization only (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.779 -> 0.792 | Hurt |
| combine full visual guidance with localized objects (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.786 -> 0.785 | Helped |
| combine full visual guidance with scene-level localization (ablation) | cyclic binaural generation · FAIR-Play split1 | STFT: 0.786 -> 0.783 | Helped |
| add the SSM module to binaural audio localization (ablation) | cyclic framework · FAIR-Play testset | Distance: 0.024 -> 0.020 | Helped |
Surprising findings
- Scene-level loss alone reaches nearly 100% localization accuracy, nearly matching the combined model.
Limitations
- Localization is unsupervised, which loses object information and can associate sounds with backgrounds.
- Limited to region-level localization; bounding-box or supervised extensions are left to future work.
Experimental setup
Datasets: FAIR-Play YT-Music
Benchmarks: FAIR-Play YT-Music
Metrics: STFT distance ENV distance WAV L2 AMP L2 PHA L2 SNR localization distance localization accuracy
Baselines: L2BNet MONO2BINAURAL APNet Sep-stereo Main network Complete network SAGM Mono-Mono
Builds on: denoising diffusion probabilistic models Mono2Binaural Sep-Stereo SAGM
Closest papers by meaning
- Seeing Sound, Hearing Sight: Uncovering Modality Bias and Conflict of AI models in Sound Localization (NeurIPS 2025 · similarity 0.87)
- Supervising Sound Localization by In-the-wild Egomotion (CVPR 2025 · similarity 0.87)
- Object-aware Sound Source Localization via Audio-Visual Scene Understanding (CVPR 2025 · similarity 0.86)
- Prompting Segmentation with Sound Is Generalizable Audio-Visual Source Localizer (AAAI 2024 · similarity 0.86)
- AV-GS: Learning Material and Geometry Aware Priors for Novel View Acoustic Synthesis (NeurIPS 2024 · similarity 0.86)
- Sounding that Object: Interactive Object-Aware Image to Audio Generation (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.