Research map / Vision-language and multimodal / Cross-modal learning
Visual Scene Graphs for Audio Source Separation
Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja, Anoop Cherian
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add the mutual-orthogonality loss on sub-graph embeddings (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 7.37 -> 8.75, SIR 13.28 -> 14.06, SAR 11.56 -> 12.95 | Improved |
| add the consistency (multi-label classification) loss (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 6.4 -> 8.75, SIR 11.2 -> 14.06, SAR 11.7 -> 12.95 | Improved |
| add the co-separation loss (ablation) | AVSGS · ASIW test | SDR / SIR (dB): SDR 1.1 -> 8.75, SIR 1.3 -> 14.06 | Improved |
| add the co-separation loss (effect on artifacts) (ablation) | AVSGS · ASIW test | SAR (dB): SAR 13.8 -> 12.95 | Hurt |
| increase the number of principal objects per video from 2 to 3 (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 8.75 -> 8.4, SIR 14.06 -> 13.5, SAR 12.95 -> 12.2 | No clear effect |
| add skip connections in the audio separator network (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 2.8 -> 8.75, SIR 4.6 -> 14.06, SAR 11.3 -> 12.95 | Improved |
| add the graph attention convolution (GATConv) on the scene graph (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 6.5 -> 8.75, SIR 11.6 -> 14.06, SAR 11.8 -> 12.95 | Improved |
| add the edge convolution (EdgeConv) on the scene graph (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 6.2 -> 8.75, SIR 10.1 -> 14.06, SAR 13.2 -> 12.95 | Improved |
| add the GRU for autoregressive sub-graph embeddings (ablation) | AVSGS · ASIW test | SDR / SIR / SAR (dB): SDR 6.49 -> 8.75, SIR 12.33 -> 14.06, SAR 10.64 -> 12.95 | Improved |
| build the scene graph from multiple frames and use more context nodes at test time (vs single frame) (ablation) | AVSGS · ASIW test | separation performance | Improved |
| replace visual-context-limited baselines by scene-graph conditioning (AVSGS vs closest competitor) on MUSIC | visually guided separation · MUSIC test | SDR / SIR (dB): around 1.3 dB on SDR and 1.6 dB on SIR | Improved |
| replace closest competitor with AVSGS on ASIW | visually guided separation · ASIW test | SDR / SIR (dB): around 1.2 dB on SDR and 2 dB on SIR | Improved |
Closest papers by meaning
- Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation (NeurIPS 2022 · similarity 0.92)
- Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation (CVPR 2024 · similarity 0.88)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.87)
- Improving Audio-Visual Segmentation with Bidirectional Generation (AAAI 2024 · similarity 0.87)
- Dynamic Derivation and Elimination: Audio Visual Segmentation with Enhanced Audio Semantics (CVPR 2025 · similarity 0.87)
- Audio-Visual Instance Segmentation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
