Research map / Large language models / Language, safety and interpretability
VocalGrad: Evaluating Acoustic Perception in Audio Language Models
Koki Ryu, Hitomi Yanaka
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces VocalGrad, a benchmark for isolating acoustic perception in large audio language models by separating it from higher-level reasoning, motivated by LALM failures on emotion recognition.
Problem
Large audio language models struggle to reliably use paralinguistic and non-linguistic audio information (e.g., in emotion recognition), and existing benchmarks entangle acoustic perception with higher-level reasoning, obscuring the source of failures.
Approach
A benchmark designed to evaluate acoustic perception in audio language models on its own, disentangled from reasoning demands. Only the abstract is available, so the specific construction details are not stated.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply transformations to speech clips and ask LALMs for change direction | five LALMs · VocalGrad | accuracy (%): 94.4 (humans) vs 51.8 (best model, Kimi-Audio) | Hurt |
| remove linguistic content (use pure sine-wave clips) (ablation) | LALMs on VocalGrad · VocalGrad sine-wave subsets (pitch, volume, noise, vibration) | accuracy (%): MiMo-Audio noise 53.2 -> 64.2; none above 70% | Mixed |
| add LoRA on the LM head only (ablation) | Kimi-Audio · VocalGrad volume/noise/roughness | accuracy (%): 88-99% on volume, noise, roughness | Improved |
| add LoRA on the LM head only (ablation) | Kimi-Audio · VocalGrad distortion/vibration/pitch/echo/clarity | accuracy (%): 52.2 distortion, 53.3 vibration, 57.0 pitch | No clear effect |
| apply LoRA to all linear layers instead of the LM head only (ablation) | Kimi-Audio · VocalGrad (vibration) | accuracy (%): 53.3 -> 99.2 | Improved |
| apply LoRA to all linear layers instead of the LM head only (ablation) | Kimi-Audio · VocalGrad (speed) | accuracy (%): 73.2 -> 99.0 | Improved |
| fine-tune on one category, test on other categories | Kimi-Audio LoRA · VocalGrad cross-category | accuracy (%) | Mixed |
| use rule-based parsing instead of exact match (ablation) | Gemini 3 Flash evaluation · VocalGrad parsing | parsed output rate (%): 83.9 -> 99.1 | Improved |
| change temporal trajectory type of the transformation (ablation) | LALMs · VocalGrad by trajectory | accuracy (%) | No clear effect |
| increase transformation intensity tier (ablation) | LALMs · VocalGrad background noise | accuracy (%) | Improved |
| decrease transformation intensity tier (ablation) | Human annotators · VocalGrad voice pitch | accuracy (%) | Hurt |
Limitations
- Only the abstract was available; full methodology and results not verified.
Experimental setup
Benchmarks: VocalGrad
Closest papers by meaning
- Listen, Think, and Understand (ICLR 2024 · similarity 0.89)
- MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark (ICLR 2026 · similarity 0.88)
- Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks (ICLR 2025 · similarity 0.88)
- Audio Large Language Models Can Be Descriptive Speech Quality Evaluators (ICLR 2025 · similarity 0.87)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.87)
- Measuring Audio's Impact on Correctness: Audio-Contribution-Aware Post-Training of Large Audio Language Models (ICLR 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
