Research map / Large language models / Language, safety and interpretability
Teaching Physical Awareness to LLMs through Sounds
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ACORN, which teaches LLMs physical awareness from sound by training a magnitude-plus-phase audio encoder on simulator-generated AQA-PHY data (1M audio-QA tuples) and connecting it to Llama3.1-8B and Qwen2-7B.
Problem
LLMs lack physical awareness of real-world acoustic phenomena such as Doppler shift, multipath, line-of-sight paths, and spatial direction, and collecting and annotating real recordings for these is costly and largely infeasible.
Approach
Decompose received sound into source and channel, synthesize training audio by convolving AudioSet sources with a physics-based channel simulator that gives exact physical labels, and encode both magnitude and sin/cos phase with a Whisper-initialized audio encoder connected to an LLM via LoRA.
Key results
- Our encoder with Qwen2-7B reaches 0.924 LOS detection BCA, 0.181 Doppler MAE_f, and 0.907 DoA MAE_t in the merged setting.
- Range estimation REP drops from 12.572 (Whisper + Llama3.1-8B) to 1.764 (ACORN + Llama3.1-8B).
- Zero-shot real-world vehicle tests give 0.870 LOS accuracy and 0.925 left/right DoA accuracy with the Qwen-based model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace Whisper with the ACORN audio encoder | Qwen2-7B · LOS Detection (Merged) | BCA: 0.881 -> 0.924 | Helped |
| Replace Whisper with the ACORN audio encoder | Qwen2-7B · Doppler Estimation (Merged) | MAE_f: 1.042 -> 0.181 | Helped |
| Replace Whisper with the ACORN audio encoder | Qwen2-7B · DoA Estimation (Merged) | MAE_t: 2.716 -> 0.907 | Helped |
| Replace Whisper with the ACORN audio encoder | Qwen2-7B · Multipath Analysis (Merged) | TCA: 0.848 -> 0.903 | Helped |
| Replace Whisper with the ACORN audio encoder | Qwen2-7B · Range Estimation (Merged) | REP: 10.609 -> 1.599 | Helped |
| Use LoRA rank 4 instead of rank 8 (ablation) | Our Encoder + Qwen2-7B · LOS Detection (Merged) | BCA: 0.924 -> 0.919 | Hurt |
| Use LoRA rank 16 instead of rank 8 (ablation) | Our Encoder + Qwen2-7B · Doppler Estimation (Merged) | MAE_f: 0.181 -> 0.193 | Hurt |
| Use LoRA rank 32 instead of rank 8 (ablation) | Our Encoder + Qwen2-7B · DoA Estimation (Merged) | MAE_t: 0.907 -> 0.731 | Helped |
| Include LOS paths in the audio (ablation) | ACORN · Multipath Analysis | TCA: 0.901 -> 0.880 (Llama); 0.912 -> 0.895 (Qwen) | Hurt |
| Introduce Doppler shifts (ablation) | ACORN · LOS Detection | BCA: 0.930 -> 0.910 (Llama); 0.936 -> 0.912 (Qwen) | Hurt |
| Increase SNR from low (<10dB) to high (>40dB) (ablation) | ACORN with Llama · Range Estimation | MAE: 5.64 samples -> 0.91 samples | Helped |
| Increase SNR from low (<10dB) to high (>40dB) (ablation) | ACORN with Qwen · Range Estimation | MAE: 5.33 samples -> 0.80 samples | Helped |
Surprising findings
- Doppler shift slightly reduces LOS detection accuracy, and LOS presence reduces multipath analysis accuracy, which the authors attribute to direct-path signals masking reverberation.
Limitations
- Model is limited to single-turn interactions.
- Real-world experiments are limited in scope and scale, with only LOS detection and left/right DoA tested.
- Results on real data show performance drop compared with simulated data.
Experimental setup
Datasets: AudioSet AQA-PHY
Benchmarks: AQA-PHY test tasks (LOS detection, Doppler estimation, DoA estimation, multipath analysis, range estimation) real-world vehicle cabin recordings (NIO ES6)
Metrics: binary classification accuracy MAE triple-class accuracy relative error percentage
Baselines: Whisper encoder with Llama3.1-8B Whisper encoder with Qwen2-7B
Builds on: Whisper LoRA MS-SWIFT AudioSet
Closest papers by meaning
- BAT: Learning to Reason about Spatial Sounds with Large Language Models (ICML 2024 · similarity 0.84)
- Listen, Think, and Understand (ICLR 2024 · similarity 0.84)
- Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities (ICML 2025 · similarity 0.83)
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (AAAI 2024 · similarity 0.83)
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (ICLR 2025 · similarity 0.83)
- OWL : Geometry-Aware Spatial Reasoning for Audio Large Language Models (ICLR 2026 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.