Research map / Vision-language and multimodal / Human and embodied modeling
Learning Spatially-Aware Language and Audio Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ELSA, a spatially-aware audio-text contrastive embedding model trained on spatially augmented AudioCaps/Clotho/Freesound with LLM-rephrased spatial captions, aligning semantic and 3D spatial attributes of audio with language.
Problem
Existing audio-language models (CLAP) are trained on non-spatial audio and cannot represent spatial attributes, while SELD models classify a fixed set of sound classes and localize to absolute coordinates rather than natural-language positions.
Approach
Spatially augment audio and captions via simulated rooms (FOA encoding) and LLM rephrasing, then train a dual-branch audio encoder (HTSAT semantic branch plus intensity-vector CNN spatial branch) with RoBERTa text encoder under CLIP loss plus direction, distance, and room-area regression losses.
Key results
- ELSA reaches 14.97 degrees MAE on REAL TUT Sound Events 2018 localization versus 95.29 for LAION-CLAP and 26.6 for SeldNET.
- ELSA achieves 44.2 AudioCaps mAP@10 versus 43.8 for LAION-CLAP.
- Zero-shot spatial attribute classification reaches 92-100% on synthetic test sets for distance, direction, and elevation, but drops to 35.8-72.1% on the real-world dataset.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ELSA instead of LAION-CLAP | AudioCaps retrieval · AudioCaps | mAP@10: 43.8 -> 44.2 | Helped |
| Use ELSA instead of SeldNET | 3D source localization · TUT Sound Events 2018 REAL | 3D localization MAE (°): 26.6 -> 14.97 | Helped |
| Use ELSA instead of PILOT | 3D source localization · TUT Sound Events 2018 REAL | 3D localization MAE (°): 4.2 -> 14.97 | Hurt |
| Train on mixed spatial and non-spatial audio and captions rather than spatial-only audio and captions (ablation) | ELSA · AudioCaps and Clotho | retrieval R@1: AudioCaps text-to-audio: 25.3 -> 33.2; Clotho audio-to-text: 12.1 -> 20.1 | Helped |
| Train on mixed spatial and non-spatial audio and captions rather than spatial audio with non-spatial captions (ablation) | ELSA · AudioCaps and Clotho | retrieval R@1: AudioCaps text-to-audio: 27.1 -> 33.2; Clotho audio-to-text: 12.4 -> 20.1 | Helped |
| Add spatial regression losses to the CLIP objective (ablation) | ELSA · Spatial audio localization and semantic retrieval | 3D localization: 27.4 -> 23.2 | Helped |
| Add spatial regression losses to the CLIP objective (ablation) | ELSA · Spatial audio | distance estimation: 54.31 -> 47.71 | Helped |
| Add spatial regression losses to the CLIP objective (ablation) | ELSA · Spatial audio retrieval | mAP@10: 24.93 -> 24.81 | Hurt |
| Use a learned spatial encoder instead of static intensity vectors (ablation) | ELSA · Spatial AudioCaps + Clotho | 3D localization MAE (°): 27.86 -> 23.2 | Helped |
| Use a learned spatial encoder instead of static intensity vectors (ablation) | ELSA · Spatial AudioCaps + Clotho | mAP@10: 23.43 -> 24.81 | Helped |
Surprising findings
- Mixing spatial and non-spatial audio is needed; training only on spatial audio/captions reduces non-spatial retrieval (e.g., 33.2 vs 27.1 R@1 on AudioCaps text-to-audio).
- Adding the spatial regression losses costs negligible semantic retrieval (0.4%) while improving localization.
Limitations
- Spatial captions from LLM rewriting can hallucinate, with mitigation left for future work.
- Real-world evaluation set is small (70 samples in five rooms) and performance drops on transfer.
- Only static sources and first-order ambisonics are handled; overlapping and moving sources are future work.
Experimental setup
Datasets: AudioCaps Clotho Freesound AudioSet Spatial LibriSpeech TUT Sound Events 2018 LAION-Audio-630K
Benchmarks: AudioCaps Clotho TUT Sound Events 2018 Spatial-AudioCaps Spatial-Clotho Spatial-RWD
Metrics: R@1 R@5 R@10 mAP@10 mean absolute error (degrees) zero-shot accuracy SPIDEr FENSE
Baselines: LAION-CLAP SeldNET PILOT Spatial LibriSpeech model CLAP (local)
Builds on: LAION-CLAP CLIP HTSAT SALSA
Closest papers by meaning
- The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.86)
- Hear, Localize, and Reason: Spatially Aware Scene Understanding for Audio-visual LLMs (NeurIPS 2026 · similarity 0.84)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.84)
- SING: Spatial Context in Large Language Model for Next-Gen Wearables (ICML 2025 · similarity 0.84)
- ADIFF: Explaining audio difference using natural language (ICLR 2025 · similarity 0.83)
- MATS: An Audio Language Model under Text-only Supervision (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.