Research map / Vision-language and multimodal / Multimodal LLMs

Real-Time Visual Attribution Streaming in Thinking Model

ICML 2026 · Vision-language and multimodal › Multimodal LLMs · approach: Chain-of-thought multimodal reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes vStream, an amortized linear estimator over cached text-to-vision attention features that predicts region-level causal ablation effects in reasoning VLMs, enabling real-time visual attribution streaming during generation.

Problem

Verifying whether long multimodal reasoning traces are grounded in visual evidence requires token-level causal attribution, but faithful causal methods (perturbation, gradient) are too costly for long traces, while raw attention is cheap but not causally faithful.

Approach

Train a lightweight linear estimator (one weight per layer-head) that maps span-pooled, region-pooled attention features to counterfactual attention-masking effects, using DINOv3-based semantic region unitization and Pearson-correlation loss, and run it asynchronously during decoding.

Key results

• vStream achieves average LDS 0.72 and Top-5 Drop 1.00 on Qwen3-VL-8B-Thinking, versus 0.71/0.89 for the strongest baseline AttnLRP.
• vStream attribution cost is about 0.024 s per 10 tokens versus about 2.6-2.8 s for gradient-based baselines, reported as up to 117x speedup.
• Cross-task transfer retains roughly 75-90% of in-domain LDS; mixing all categories recovers full performance.

Surprising findings

• Raw attention alone is far less faithful (average LDS 0.40 on Qwen3-VL) than the learned estimator, despite being the feature source.
• Unsuccessful reasoning chains show a fixation pattern for hallucinations and wandering for reasoning errors, with failure prediction AUC 0.69 at 30% of reasoning.

Limitations

• Linear estimator assumes attention features linearly predict ablation effects, an empirical assumption without theoretical guarantee.
• Estimators are trained separately per model and do not transfer across architectures; cross-task transfer weaker for Document tasks.

Experimental setup

Datasets: GQA DocVQA ScienceQA MathVista MathVision MathVerse ChartMimic WebSight

Benchmarks: GQA DocVQA ScienceQA MathVista MathVision MathVerse ChartMimic WebSight POPE CHAIR

Metrics: LDS (Linear Datamodeling Score) Top-5 Drop R^2 latency (s/10 tokens)

Baselines: Random Attention InputGrad AttnLRP TAM

Builds on: DINOv3 Ward's linkage agglomerative clustering TAM AttnLRP InputGrad

Closest papers by meaning

• VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning (ICML 2026 · similarity 0.87)
• Look on Demand: A Cognitive Scheduling Framework for Visual Evidence Acquisition in Multimodal Reasoning (ICML 2026 · similarity 0.87)
• From Narrow to Panoramic Vision: Attention-Guided Cold-Start Reshapes Multimodal Reasoning (ICLR 2026 · similarity 0.86)
• Perception Without Engagement: Dissecting the Causal Discovery Deficit in LMMs (NeurIPS 2026 · similarity 0.86)
• Visual Thoughts: A Unified Perspective of Understanding Multimodal Chain-of-Thought (NeurIPS 2025 · similarity 0.86)
• From Prompts to Tokens: Internalizing Causal Supervision in Vision-Language Model for Multi-Image Causal Reasoning (ICML 2026 · similarity 0.86)