Research map / Vision-language and multimodal / Cross-modal learning
OmniSep: Unified Omni-Modality Sound Separation with Query-Mixup
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OmniSep, a query-based sound separation model that accepts text, image, audio, and composed multi-modal queries via ImageBind features, trained with Query-Mixup, with negative queries and Query-Aug for open-vocabulary text.
Problem
Existing query-based sound separation methods use only one query modality, cannot combine composed multi-modal queries, cannot remove sounds via undesired-sound queries, and fail on open-vocabulary text.
Approach
Mix frozen ImageBind query embeddings from different modalities during training (Query-Mixup) so one separator optimizes all modalities jointly; subtract weighted negative-query embeddings at inference (training-free) to remove unwanted sounds; retrieve the nearest in-domain class query for free-form text (Query-Aug).
Key results
- OmniSep improves mean SDR over prior methods across TQSS, IQSS and AQSS, e.g. 10.65 vs 9.82 (AudioSEP) for text queries on MUSIC and 6.70 vs 6.26 on VGGSOUND-CLEAN+.
- Adding negative queries (alpha=0.5) raises VGGSOUND-CLEAN+ text-queried mean SDR from 6.70 to 7.57 and MUSIC-CLEAN+ from 12.55 to 14.15.
- Query-Aug raises mean SDR on unrestricted text descriptions from 4.95 to 6.32 on VGGSOUND-CLEAN+.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add negative query (ablation) | OmniSep · MUSIC TQSS | Mean SDR: 10.65$\pm$1.07 -> 10.92$\pm$1.06 | Helped |
| Add negative query (ablation) | OmniSep · VGGSOUND-CLEAN+ TQSS | Mean SDR: 6.70$\pm$0.66 -> 7.57$\pm$0.67 | Helped |
| Add negative query (ablation) | OmniSep · MUSIC-CLEAN+ TQSS | Mean SDR: 12.55$\pm$0.77 -> 14.15$\pm$0.95 | Helped |
| Add negative query (ablation) | OmniSep · MUSIC IQSS | Mean SDR: 10.97$\pm$1.03 -> 11.08$\pm$1.02 | Helped |
| Add negative query (ablation) | OmniSep · VGGSOUND-CLEAN+ IQSS | Mean SDR: 6.69$\pm$0.67 -> 7.68$\pm$0.69 | Helped |
| Add negative query (ablation) | OmniSep · MUSIC-CLEAN+ IQSS | Mean SDR: 13.15$\pm$0.92 -> 13.78$\pm$1.02 | Helped |
| Add negative query (ablation) | OmniSep · MUSIC AQSS | Mean SDR: 10.26$\pm$1.13 -> 10.40$\pm$1.11 | Helped |
| Add negative query (ablation) | OmniSep · VGGSOUND-CLEAN+ AQSS | Mean SDR: 7.12$\pm$0.65 -> 7.22$\pm$0.68 | Helped |
| Add negative query (ablation) | OmniSep · MUSIC-CLEAN+ AQSS | Mean SDR: 12.92$\pm$0.89 -> 13.43$\pm$0.98 | Helped |
| Add negative query (ablation) | OmniSep · VGGSOUND-CLEAN+ composed omni-modal | Mean SDR: 7.46$\pm$0.65 -> 8.00$\pm$0.69 | Helped |
| Train jointly on text and image queries instead of text-only (ablation) | OmniSep training configuration (#1 to #3) · TQSS and IQSS; AVG SDR | AVG SDR: 5.12$\pm$0.59 -> 6.42$\pm$0.72 | Helped |
| Add audio modality training data to text-and-image joint training (ablation) | OmniSep training configuration (#3 to #4) · TQSS and IQSS; AVG SDR | AVG SDR: 6.42$\pm$0.72 -> 6.45$\pm$0.68 | Helped |
Surprising findings
- Naive direct subtraction for negative queries is highly sensitive to the weight alpha, while the proposed proportional weighting varies by no more than 0.45 SDR.
- Single-modality-trained models can outperform the joint model on their own modality (e.g., text-only training gives 0.73 higher text-queried mean SDR than the joint model), though Query-Mixup narrows the gap.
Limitations
- Evaluation limited to VGGSOUND and MUSIC datasets, which do not cover all real-world sound events such as pop music.
- Negative-query weight alpha still needs selection, and Query-Aug depends on the in-domain query set.
- Relies on the large-scale ChatGPT-generated rewrites for open-vocabulary evaluation.
Experimental setup
Datasets: VGGSOUND MUSIC VGGSOUND-CLEAN+ MUSIC-CLEAN+
Benchmarks: MUSIC VGGSOUND-CLEAN+ MUSIC-CLEAN+
Metrics: SDR (mean) SDR (median)
Baselines: LabelSep PIT BERTSep CLIPSEP-NIT CLIPSEP CLIPSEP-Text AudioSEP SOP i-Query AQSS
Builds on: CLIPSEP ImageBind AudioSEP CLIP
Closest papers by meaning
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.87)
- Independency Adversarial Learning for Cross-Modal Sound Separation (AAAI 2024 · similarity 0.86)
- MGE-LDM: Joint Latent Diffusion for Simultaneous Music Generation and Source Extraction (NeurIPS 2025 · similarity 0.84)
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.84)
- BNMusic: Blending Environmental Noises into Personalized Music (NeurIPS 2025 · similarity 0.84)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.