Research map / Vision-language and multimodal / Multimodal LLMs

To Think or Not to Think: Pre-Decisional Reasoning Budgets for Referring Audio-Visual Segmentation

NeurIPS 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that generation-onset hidden states of a frozen Ref-AVS MLLM linearly predict whether extra chain-of-thought reasoning helps, and uses this single-forward-pass signal to route each query to ZERO/SHORT/LONG reasoning budgets, cutting reasoning tokens by 60% while keeping about 96% of always-long J&F.

Problem

Chain-of-thought reasoning before grounding in referring audio-visual segmentation (Ref-AVS) is assumed to always help, but it can hurt simple queries ("overthinking trap") while costing many tokens.

Approach

Reads the final-layer hidden state at the last input token (before the first reasoning token) with a linear probe or a 2-layer MLP to predict a counterfactually labeled reasoning budget, and uses that state as a single-pass router that selects ZERO, SHORT, or LONG reasoning before generation.

Key results

• Linear probe on the generation-onset state predicts binary reasoning need with 70.1% accuracy vs. 57.7% majority baseline and 58.1% text-only features.
• The onset router reaches 0.528 J&F on RefAVSBench test_s at 110 tokens vs. 0.549 J&F and 275 tokens for Always-Long (about 96% retention, 60% token reduction).
• Compared with Short-then-Decide, the onset router reduces reasoning tokens by 39.6% with a 0.007 J&F drop; it has a 0.820 s vs. 2.13 s mean decision time on an A100.

Surprising findings

• Forced long chain-of-thought can degrade segmentation on simple queries; on audited harmful-long cases J&F falls by 16.5 points.
• Source-only transfer to R2-AVSBench gives 0.458 J&F, below Always-Long's 0.474, so the OOD benefit is cost-efficiency rather than accuracy.

Limitations

• CBL labels require target masks and pipeline-specific counterfactual outcomes, so backbone, prompt, or distribution changes may need recalibration.
• Hand-crafted proxy comparison does not establish irreducibility or a causal self-assessment mechanism.
• Router results are five-fold out-of-fold estimates within the benchmark populations rather than evaluation on an untouched official test split.

Experimental setup

Datasets: RefAVSBench R2-AVSBench

Benchmarks: RefAVSBench test_s RefAVSBench test_u R2-AVSBench test_s

Metrics: J&F J F tokens binary accuracy balanced accuracy latency

Baselines: Always-Zero Always-Short Always-Long (TGS-Agent) Text Router (PBS-A) Multimodal Mean Pool Confidence Router Short-then-Decide

Builds on: TGS-Agent SAM2 GroundingDINO MAR3 LoRA

Closest papers by meaning

• Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation (AAAI 2026 · similarity 0.93)
• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation (AAAI 2026 · similarity 0.90)
• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning (ICML 2026 · similarity 0.90)
• AStar: Boosting Multimodal Reasoning with Automated Structured Thinking (AAAI 2026 · similarity 0.90)
• See It, Say It, Sorted: An Iterative Training-Free Framework for Visually-Grounded Multimodal Reasoning in LVLMs (CVPR 2026 · similarity 0.89)
• DiffThinker: Towards Generative Multimodal Reasoning with Diffusion Models (ICML 2026 · similarity 0.89)