Research map / Label-efficient and robust learning / Self-supervised and contrastive
SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSLAM, a self-supervised audio pre-training method that trains a masked latent bootstrapping model on partially max-mixed AudioSet spectrograms with a source retention loss, improving polyphonic and standard audio benchmark performance.
Problem
Self-supervised audio encoders are widely used frozen in multimodal LLMs, but are evaluated mostly on monophonic benchmarks, so their ability to handle polyphonic real-world audio is underexplored.
Approach
Use audio mixtures as an SSL pretext in a masked latent bootstrapping framework: partially mix spectrograms via element-wise max, distill from teacher features of the constituent sources, and add a source retention loss that aligns the mixed-audio student output with the average of its source teacher representations.
Key results
- SSLAM reaches 50.2 mAP on AudioSet-2M, up to 3.9% over prior SOTA, and 40.9 mAP on AudioSet-20K under fine-tuning.
- On polyphonic datasets SSLAM improves linear evaluation and fine-tuning by up to 9.1% mAP (SPASS Market), with a 21.6% gain on AS-20K linear evaluation.
- SSLAM achieves 98.8% on KS1 and 96.2% on ESC-50, matching or exceeding prior SOTA on monophonic benchmarks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| pre-train with SSLAM audio mixtures and source retention loss | ViT-Base · AS-2M | mAP: 48.6 -> 50.2 | Helped |
| pre-train with SSLAM audio mixtures and source retention loss | ViT-Base · AS-20K | mAP: 40.2 -> 40.9 | Helped |
| pre-train with SSLAM audio mixtures and source retention loss | ViT-Base · ESC-50 | accuracy: 96.3 -> 96.2 | No clear effect |
| pre-train with SSLAM audio mixtures and source retention loss | ViT-Base · KS2 | accuracy: 98.9 -> 98.1 | Hurt |
| pre-train with SSLAM audio mixtures and source retention loss | ViT-Base · KS1 | accuracy: 98.2 -> 98.8 | Helped |
| add partially mixed audio objectives instead of unmixed-only objectives (ablation) | MB-UA · SPASS and other polyphonic datasets | mAP: 62.8 -> 67.4 | Helped |
| use unmixed and partially mixed audio objectives instead of mixed-only objectives (ablation) | MB-PMA · Polyphonic datasets | mAP | Hurt |
| add source retention loss to unmixed and partially mixed objectives (ablation) | MB-UA-PMA · Polyphonic datasets | mAP: 67.4 -> 68.5 | Helped |
| add global loss on unmixed data (ablation) | local-loss training objectives · AS-20K | mAP: 40.4 -> 40.7 | Helped |
| add global loss on partially mixed data (ablation) | local-loss training objectives with unmixed global loss · AS-20K | mAP: 40.7 -> 40.9 | Helped |
| add SRL to the global and local losses (ablation) | SSLAM objectives · AS-20K | mAP: 40.9 -> 40.6 | Hurt |
| use partial instead of full spectrogram mixing (ablation) | SSLAM · AS-20K | mAP: 39.0 -> 40.6 | Helped |
Surprising findings
- Mixing only a portion of each clip (partial mixing) worked better than full mixing in the stage-2 ablation (40.6 vs 40.4 mAP fine-tuning).
- Adding the global loss on the final teacher layer only helped, but adding SRL to the global loss slightly reduced AS-20K performance (40.9 to 40.6).
Limitations
- Polyphonic evaluation relies on synthetic datasets (SPASS, URBAN-SED, IDMT-DESED-FL) rather than real recordings.
- Mixing-based pre-training halves the unmixed data per batch, trading off some polyphonic gains against monophonic performance in MB-UA-PMA.
- Concept separation via MixIT alternative was less effective (39.9 vs 40.9 mAP on AS-20K).
Experimental setup
Datasets: AudioSet-2M AudioSet-20K ESC-50 KS1 KS2 SPASS IDMT-DESED-FL URBAN-SED
Benchmarks: AudioSet-2M AudioSet-20K ESC-50 KS1 KS2 SPASS IDMT-DESED-FL URBAN-SED Degrees of Polyphony
Metrics: mAP accuracy
Baselines: SS-AST MAE-AST MaskSpec MSM-MAE data2vec Audio-MAE BEATs ASiT A-JEPA EAT
Builds on: Audio-MAE data2vec BEATs ASiT A-JEPA
Closest papers by meaning
- MERT: Acoustic Music Understanding Model with Large-Scale Self-supervised Training (ICLR 2024 · similarity 0.89)
- Weakly-supervised Audio Separation via Bi-modal Semantic Similarity (ICLR 2024 · similarity 0.86)
- Massive Sound Embedding Benchmark (MSEB) (NeurIPS 2025 · similarity 0.85)
- CompA: Addressing the Gap in Compositional Reasoning in Audio-Language Models (ICLR 2024 · similarity 0.85)
- CA-SSLR: Condition-Aware Self-Supervised Learning Representation for Generalized Speech Processing (NeurIPS 2024 · similarity 0.85)
- AudioCALM: Continuous Autoregressive Language Modeling for Universal Audio Generation (NeurIPS 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.