Research map / Vision-language and multimodal / Multimodal LLMs

Senses Wide Shut: A Representation-Action Gap in Omnimodal LLMs

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Unified any-modality language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces IMAVB, a 500-clip movie benchmark in a 2x2 design (vision/audio x standard/misleading premise), finds a representation-action gap in omnimodal LLMs (probes decode premise mismatch but models rarely reject), and proposes probe-guided logit adjustment (PGLA) as a diagnostic intervention.

Problem

Omnimodal LLMs often accept textual premises that contradict their own video/audio input; it is unclear whether this failure is one of perception (mismatch not encoded) or action (encoded mismatch not expressed in output).

Approach

Shows that linear probes on hidden states decode premise-perception mismatches (more strongly for vision than audio) even when output rejection is near zero, and that re-injecting the probe signal into output logits via a confidence-gated adjustment improves balanced rejection accuracy.

Key results

• Seven of eight open-source models under-reject: misleading vision detection <=16.2% and audio <=6.6% under fixed order, while handling 40-75% of standard questions.
• Modality-specific linear probes recover the misleading premise from hidden states at up to 86% on vision, with residualized margins of 9-15pp internal gap across six 7B models.
• PGLA improves mean balanced accuracy by +9.9pp at a 3pp standard-accuracy budget and +15.0pp unconstrained across eight open-source models.

Surprising findings

• Qwen3-Omni and Gemini 3.1 Pro over-reject (72.8% and 94.0% on misleading vision) by trading standard accuracy, rather than genuinely grounding.
• Vanilla logit lens shows near-zero signal for several models even though learned probes decode the misleading signal with 83-99% accuracy.

Limitations

• Gemini 3.1 Pro evaluated only on baseline experiment due to API cost; no representational analysis for proprietary model.
• Judge (Qwen3.5-27B) shares model family with QA generator, so judge results are partial evidence only.
• PGLA is a diagnostic intervention that shifts decision boundary toward rejection and reduces standard accuracy for most models.

Experimental setup

Datasets: IMAVB

Benchmarks: IMAVB

Metrics: accuracy balanced accuracy probe accuracy residualized probe accuracy logit lens P(correct)

Baselines: Gemini 3.1 Pro TF-IDF text probe sentence-BERT text probe AVHBench

Builds on: Inference-Time Intervention Representation Engineering logit lens lmms-eval

Closest papers by meaning

• A Benchmark for Omni-Modal Reasoning in Long Videos (NeurIPS 2026 · similarity 0.89)
• XModBench: Benchmarking Cross-Modal Capabilities and Consistency in Omni-Language Models (ICLR 2026 · similarity 0.89)
• OmniVideoBench: Towards Audio-Visual Understanding Evaluation for Omni MLLMs (ICLR 2026 · similarity 0.89)
• Sandboxed Coding Agents are Competitive Omni-modal Task Solvers (NeurIPS 2026 · similarity 0.88)
• Omni-Captioner: Data Pipeline, Models, and Benchmark for Omni Detailed Perception (ICLR 2026 · similarity 0.88)
• From Senses to Decisions: The Information Flow of Auditory and Visual Perception in Multimodal LLMs (NeurIPS 2026 · similarity 0.88)