Research map / Vision-language and multimodal / Multimodal LLMs
ANCHOR: Audio-Visually Grounded Chain-of-Thought Reasoning Benchmark
Joel Julin, Souraja Kundu, Liza Dahiya, George Z Wei, Louise Xie, Jingru Yi, Xu Zhang, Hao Yang, László Jeni
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces ANCHOR, a benchmark for evaluating whether multimodal LLMs ground their audio-visual chain-of-thought reasoning in the actual visual regions and audio events, rather than only checking final answer accuracy.
Problem
MLLMs score well on audio-visual question answering, but correct answers may come from linguistic priors or shortcuts rather than perceptually grounded reasoning; existing benchmarks mostly measure final-answer accuracy and give little insight into the reasoning process.
Approach
A benchmark that evaluates chain-of-thought reasoning for audio-visual questions by checking grounding of reasoning steps to specific visual regions and audio events, not only the final answer. Only the abstract is available, so details of the construction and metrics are unknown.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| remove video (audio-only input) (ablation) | Gemini-3-Flash · ANCHOR human-refined set | accuracy (%): 57.7 -> 16.1 | Hurt |
| remove video (audio-only input) (ablation) | GPT-5.6-Luna · ANCHOR human-refined set | accuracy (%): 52.5 -> 10.7 | Hurt |
| remove video (audio-only input) (ablation) | Claude-Sonnet-5 · ANCHOR human-refined set | accuracy (%): 52.6 -> 14.2 | Hurt |
| add audio evidence (ablation) | Gemini-3-Flash · ANCHOR human-refined set | accuracy (%): 52.9 -> 57.7 | Improved |
| add bounding boxes (ablation) | Gemini-3-Flash · ANCHOR human-refined set | accuracy (%): 50.0 -> 57.7 | Improved |
| add bounding boxes (ablation) | Claude-Sonnet-5 · ANCHOR human-refined set | accuracy (%): 49.4 -> 52.6 | Improved |
| add audio and bounding boxes together (ablation) | GPT-5.6-Luna · ANCHOR human-refined set | accuracy (%): 45.0 -> 52.5 | Improved |
| compare references: human-refined vs Gemini- or GPT-generated reference (ablation) | model ranking · ANCHOR focused subset (100 videos, 446 QA) | Pearson r between rankings: r=0.96-1.00, mean 0.99 | No clear effect |
| use models with comparable answer score (Gemini-3-Flash vs GPT-5.2) | CoT evaluation · ANCHOR full benchmark | Hallucination/faithfulness: Answer 3.83 vs 3.77; Halluc. 2.20 vs 2.05 | Mixed |
| longer reasoning output (more tokens) | MLLM CoT · ANCHOR full benchmark | answer and faithfulness: Grok-4.1-Fast 644 tokens: Answer 3.39, Halluc. 1.48 | No clear effect |
Experimental setup
Benchmarks: ANCHOR
Closest papers by meaning
- OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.91)
- LENS: Multi-level Evaluation of Multimodal Reasoning with Large Language Models (ICLR 2026 · similarity 0.90)
- HAVE-Bench: Hierarchical Audio-Visual Evaluation from Perception to Interaction (CVPR 2026 · similarity 0.90)
- Mellow: a small audio language model for reasoning (NeurIPS 2025 · similarity 0.90)
- CG-Bench: Clue-grounded Question Answering Benchmark for Long Video Understanding (ICLR 2025 · similarity 0.89)
- MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
