Research map / Large language models / Language, safety and interpretability

SING: Spatial Context in Large Language Model for Next-Gen Wearables

ICML 2025 · Large language models › Language, safety and interpretability · approach: Speech and audio language models

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds SING, a wearable LLM system that fuses microstructure-based monaural direction-of-arrival (DoA) embeddings with Whisper speech embeddings, projects them into LLaMA-3.2 3B via a linear layer, and fine-tunes with LoRA for spatially aware ASR and multi-speaker soundscaping.

Problem

Existing spatial-audio LLMs (e.g., BAT) handle non-speech sounds with large microphone arrays or have high DoA error (88.52°), and speech LLMs lack spatial awareness; wearables need compact, low-power spatial speech understanding.

Approach

Combine an Owlet-style microstructure monaural sensing front-end (a CNN DoA encoder on mel-spectrograms) with Whisper-large-v3 encoder features, align the concatenated spatial and speech embeddings to the LLM input space with a LLaVA-style linear projector, and LoRA-fine-tune LLaMA-3.2 3B; a number-of-speakers encoder plus per-count DoA encoders extends this to multi-source soundscaping.

Key results

• Spatial ASR DoA MAE of 25.72° vs BAT's 88.52° for one source (monaural).
• Spatially aware ASR WER of 5.3% vs SALMONN's 2.2%; without spatial features WER is 1.8%.
• Soundscaping DoA MAE of 24.16°, 28.11°, 23.31°, 17.08° for 2 to 5 sources, with median error of 13° for one source and 13–20° for multi-source cases.

Measured effects reported in the paper

Surprising findings

• Spatial features raise WER from 1.8% to 5.3% relative to the no-spatial variant, a trade-off the authors acknowledge.
• The monaural SING model's MAE for 5 sources (17.08°) is lower than for 1 source (25.72°).

Limitations

• Evaluated only on synthetic data convolved with calibrated impulse responses; no real-world recordings.
• Higher WER than SALMONN when spatial features are included.
• Azimuth only; elevation estimation left to future work.

Experimental setup

Datasets: LibriSpeech

Benchmarks: LibriSpeech-based synthetic spatial speech dataset

Metrics: MAE median error MEEM WER

Baselines: BAT SALMONN SELDNet AudioMAE

Builds on: Owlet LLaVA Whisper LoRA

Closest papers by meaning

• MRSAudio: A Large-Scale Multimodal Recorded Spatial Audio Dataset with Refined Annotations (NeurIPS 2025 · similarity 0.87)
• The World is Not Mono: Enabling Spatial Understanding in Large Audio-Language Models (NeurIPS 2026 · similarity 0.86)
• BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.86)
• OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.86)
• STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence (ICLR 2026 · similarity 0.85)
• SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence (ICLR 2026 · similarity 0.85)