Research map / Vision-language and multimodal / Cross-modal learning
FLAM: Frame-Wise Language-Audio Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces FLAM, an open-vocabulary frame-wise audio-language contrastive model trained with a text-dependent logit-adjusted binary frame-level objective on a 1M-sample synthetic sound event detection dataset, improving localization while keeping retrieval and zero-shot classification.
Problem
Contrastive audio-language models give only clip-level alignment and cannot localize when a described sound event occurs, while traditional SED is limited to fixed label sets and frame-level paired text-audio labels are scarce.
Approach
Extend CLAP-style ALMs with frame-level embeddings and a binary frame-text contrastive loss whose logit scale and bias are text-dependent and estimated via logit adjustment to remove spurious label-imbalance correlations, trained on synthetic 10-second mixtures with automatically derived frame labels.
Key results
- FLAM reaches 91.0 AUROC on Held-out and 81.23 on ASFX-SED versus 74.17 and 69.56 for MGA-CLAP retrained on the same data.
- FLAM improves zero-shot accuracy over FLAM-Global on ESC50 (86.9 vs 81.6) and US8K (75.6 vs 65.4) while retrieval stays close to FLAM-Global on Clotho and AudioCaps.
- On DESED, MGA-CLAP reported PSDS 26.4 remains higher than FLAM's 9.37 under the paper's setup, so FLAM does not win on every PSDS metric.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add frame-wise contrastive training and prior loss | FLAM-Global · Held-out | AUROC: 67.76 -> 91.0 | Helped |
| add frame-wise contrastive training and prior loss | FLAM-Global · ASFX-SED | AUROC: 65.14 -> 81.23 | Helped |
| add frame-wise contrastive training and prior loss | FLAM-Global · DESED | PSDS: 7.09 -> 9.37 | Hurt |
| add frame-wise contrastive training and prior loss | FLAM-Global · ESC-50 | accuracy: 81.6 -> 86.9 | Helped |
| add frame-wise contrastive training and prior loss | FLAM-Global · US8K | accuracy: 65.4 -> 75.6 | Helped |
| add frame-wise contrastive training and prior loss | FLAM-Global · VGGSound | accuracy: 38.9 -> 39.3 | Helped |
| remove closed-set SED training data (ablation) | FLAM · ASFX-SED | AUROC: 81.23 -> 74.94 | Hurt |
| remove closed-set SED training data (ablation) | FLAM · UrbanSED | AUROC: 93.62 -> 76.12 | Hurt |
| remove global contrastive loss (ablation) | FLAM · ASFX-SED | AUROC: 81.23 -> 81.15 | No clear effect |
| remove global contrastive loss (ablation) | FLAM · ASFX retrieval | R@1 T2A: 4.4 -> 0.8 | Hurt |
| remove global contrastive loss (ablation) | FLAM · ESC-50 | accuracy: 86.9 -> 66.9 | Hurt |
| increase temporal resolution from L=32 to L=128 (ablation) | FLAM · ASFX-SED | AUROC: 81.23 -> 82.51 | Helped |
Surprising findings
- Removing the global contrastive loss during SED training leaves SED metrics roughly unchanged but collapses retrieval (e.g., AudioCaps T2A R@1 3.9) and zero-shot accuracy on VGGSound (18.5).
- Removing the closed-set SED datasets causes a large AUROC drop on UrbanSED (76.12 vs 93.62).
Limitations
- Fixed 10-second input and coarse frame resolution (L=32) limit handling of longer or finer temporal recordings.
- Training corpus is limited in scale and model is relatively lightweight.
- Real-world frame-level annotations are not used; PSDS on DESED is lower than the reported MGA-CLAP number.
Experimental setup
Datasets: LAION-Audio-630K AudioCaps Clotho AudioSet-Strong DESED UrbanSED MAESTRO ESC-50 UrbanSound8K VGGSound ASFX
Benchmarks: Held-out ASFX-SED DESED MAESTRO AudioSet-Strong UrbanSED Clotho AudioCaps ASFX ESC-50 UrbanSound8K VGGSound
Metrics: PSDS MPAUC AUROC recall@1 recall@5 zero-shot accuracy Spearman correlation
Baselines: FLAM-Global MGA-CLAP LAION-CLAP CompA
Builds on: CLAP SigLIP MGA-CLAP HTSAT
Closest papers by meaning
- Towards Open-Vocabulary Audio-Visual Event Localization (CVPR 2025 · similarity 0.87)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.86)
- Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing (AAAI 2025 · similarity 0.85)
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.85)
- Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding (AAAI 2025 · similarity 0.85)
- MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Tasks (ICML 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.