Research map / Large language models / Alignment and preferences
MARS-Sep: Multimodal-Aligned Reinforced Sound Separation
Zihan Zhang, Xize Cheng, Zhennan Jiang, Dongjie Fu, Jingyuan Chen, Zhou Zhao, Tao Jin
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add RL with multimodal reward on top of supervised OmniSep (frozen encoder) (ablation) | OmniSep · VGGSound-clean+ text | SDR: 6.70 -> 6.71 | Improved |
| add progressive fine-tuning of encoder with supervised loss only (ablation) | OmniSep · VGGSound-clean+ text | SDR: 6.70 -> 0.75 | Hurt |
| add RL to the fine-tuned encoder (full model) (ablation) | MARS-Sep · VGGSound-clean+ text | SDR: 0.75 -> 6.91 | Improved |
| use 3-stage instead of no fine-tuning of the reward encoder (ablation) | MARS-Sep · VGGSound-clean+ audio query | SDR: 6.85 -> 7.33 | Improved |
| use 3-stage instead of 1-stage fine-tuning (ablation) | MARS-Sep · VGGSound-clean+ image query | SDR: 6.54 -> 6.93 | Mixed |
| use MLBP query pooling instead of max pooling (ablation) | MARS-Sep · MUSIC-clean+ text | CLAP_t: 5.34 -> 6.18 | Improved |
| use MLBP instead of learned weighted sums (ablation) | MARS-Sep · VGGSound-clean+ composed query | SDR: 8.08 vs 7.93 | Mixed |
| increase Beta concentration kappa from 3 to 9 (ablation) | MARS-Sep · VGGSound-clean+ composed query | SDR: 6.99 -> 7.93 | Improved |
| vary PPO clip range epsilon 0.01-0.4 (ablation) | MARS-Sep · VGGSound-clean+ composed query | SDR: 7.90 to 7.93 | No clear effect |
| raise entropy coefficient from 0.01 to 0.2 (ablation) | MARS-Sep · VGGSound-clean+ composed query | SDR: 7.20 -> 7.93 | Improved |
| add KL penalty 0.1 vs none (ablation) | MARS-Sep · VGGSound-clean+ composed query | SDR: 7.43 -> 7.93 | Mixed |
| replace OmniSep supervised mask regression with MARS-Sep | OmniSep · MUSIC-clean+ audio query | SDR: 10.37 -> 11.73 | Improved |
Closest papers by meaning
- Continual Audio-Visual Sound Separation (NeurIPS 2024 · similarity 0.88)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.86)
- Seeing Speech and Sound: Distinguishing and Locating Audio Sources in Visual Scenes (CVPR 2025 · similarity 0.85)
- CAV-MAE Sync: Improving Contrastive Audio-Visual Mask Autoencoders via Fine-Grained Alignment (CVPR 2025 · similarity 0.85)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.85)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
