Research map / Vision-language and multimodal / Cross-modal learning
Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusion
Jian Ma, Wenguan Wang, Yi Yang, Feng Zheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add mutual learning between reverberator and dereverberator (no unpaired data) (ablation) | VSD · SoundSpaces-Speech (VAM) | STFT distance: 0.657 -> 0.573 | Improved |
| add unpaired natural and anechoic audio to mutual learning (ablation) | MVSD · SoundSpaces-Speech (VAM) | STFT distance: 0.573 -> 0.508 | Improved |
| add unpaired data to mutual learning for dereverberation (ablation) | MVSD · SoundSpaces-Speech (dereverberation) | WER (%): 4.32 -> 4.27 | Improved |
| add visual scene conditioning (ablation) | MVSD · SoundSpaces-Speech Seen (VAM) | RTE (s): 0.188 -> 0.030 | Improved |
| replace GAN-based generator with diffusion model (ablation) | MVSD · SoundSpaces-Speech (VAM) | STFT distance: 0.831 -> 0.657 | Improved |
| add diffusion process to controllable Unet (ablation) | MVSD · SoundSpaces-Speech (dereverberation) | WER (%): 6.74 -> 4.27 | Improved |
| increase denoising steps from 150 to 250 (ablation) | MVSD · SoundSpaces-Speech (VAM) | STFT distance: 1.452 -> 0.508 | Improved |
| increase denoising steps from 250 to 1000 (ablation) | MVSD · SoundSpaces-Speech (VAM) | STFT distance: 0.508 -> 0.487 | No clear effect |
| increase unpaired data from 0k to 5k (ablation) | MVSD · SoundSpaces-Speech (VAM) | STFT distance: 0.573 -> 0.508 | Improved |
| add mutual learning (MVSD) vs SOTA AViTAR on Seen split | VAM · SoundSpaces-Speech Seen (VAM) | STFT distance: 0.665 -> 0.508 | Improved |
Closest papers by meaning
- AV-RIR: Audio-Visual Room Impulse Response Estimation (CVPR 2024 · similarity 0.87)
- Resounding Acoustic Fields with Reciprocity (NeurIPS 2025 · similarity 0.86)
- From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation (ICML 2024 · similarity 0.86)
- Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners (CVPR 2024 · similarity 0.86)
- A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation (NeurIPS 2024 · similarity 0.85)
- V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models (AAAI 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
