Research map / Vision-language and multimodal / Cross-modal learning
Independency Adversarial Learning for Cross-Modal Sound Separation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes IAL-CMS, a cross-modal sound separation model that uses adversarial minimization of mutual information between separated sound elements plus audio-visual consistency learning, enabling unsupervised separation from mixtures alone.
Problem
Separating sound mixtures with heavy overlap and noise, especially in the unsupervised setting where single-source ground-truth sounds are unavailable.
Approach
Uses a GAN-style discriminator (integral probability metric, spectral normalization) to estimate and minimize mutual information between separated sound elements, reference distribution given by independent element pairs from a sound element bank; combined with audio-visual contrastive consistency and cross-attention fusion, and mixture reconstruction loss.
Key results
- On MUSIC 2-mix supervised, SDR 12.26 vs. FCSN 11.74 and Co-Separation 7.38; unsupervised variant reaches SDR 10.22.
- On VGGSound-15Instrument 2-mix, SDR 9.96 and SIR 14.74, beating FCSN (6.39 SDR) by 3.57 SDR.
- Ablation on MUSIC shows IAL (adversarial loss) adds 2.14 SDR and 3.98 SIR; removing audio-visual consistency loss drops SDR by 1.91.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add IAL losses LG/LD (ablation) | IAL-CMS · MUSIC | SDR: 8.08 -> 10.22 | Helped |
| Add IAL losses LG/LD (ablation) | IAL-CMS · MUSIC | SIR: 13.10 -> 17.08 | Helped |
| Add audio-visual consistency loss Lav (ablation) | IAL-CMS · MUSIC | SDR: 8.31 -> 10.22 | Helped |
| Add audio-visual consistency loss Lav (ablation) | IAL-CMS · MUSIC | SIR: 14.18 -> 17.08 | Helped |
| Add spatial attention (ablation) | IAL-CMS audio-visual interaction module · MUSIC | SDR: 11.01 -> 11.33 | Helped |
| Add channel attention after spatial attention (ablation) | IAL-CMS audio-visual interaction module · MUSIC | SDR: 11.33 -> 11.84 | Helped |
| Add AdaIN to the decoder (ablation) | IAL-CMS · MUSIC | SDR: 11.84 -> 12.26 | Helped |
| Use IAL-CMS instead of FCSN | sound separation · AudioSet-15Instrument 2-mix | SIR: 7.92 -> 6.70 | Hurt |
Surprising findings
- The unsupervised variant (trained on mixtures only) outperforms supervised Co-Separation and SepFusion on MUSIC.
Limitations
- Adversarial min-max training is hard to stabilize; several algorithms were tried before settling on spectral normalization.
- Slightly weaker SIR than FCSN on AudioSet-15Instrument and 3-mix MUSIC setting.
Experimental setup
Datasets: MUSIC VGGSound AudioSet Synthetic-Duet
Benchmarks: MUSIC VGGSound-15Instrument AudioSet-15Instrument
Metrics: SDR SIR SAR
Baselines: SOP MP-Net Co-Separation FCSN AVSGS SepFusion AMNet iQuery MIML
Builds on: Co-Separation MixIT Mix-and-Separate McGAN InfoNCE
Closest papers by meaning
- iQuery: Instruments As Queries for Audio-Visual Sound Separation (CVPR 2023 · similarity 0.86)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.86)
- OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup (ICLR 2025 · similarity 0.86)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.84)
- CrossMAE: Cross-Modality Masked Autoencoders for Region-Aware Audio-Visual Pre-Training (CVPR 2024 · similarity 0.83)
- Identifiable Shared Component Analysis of Unpaired Multimodal Mixtures (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.