Research map / Vision-language and multimodal / Multimodal LLMs

Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward

CVPR 2026 · Vision-language and multimodal › Multimodal LLMs · approach: VLM capability analysis · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Saliency-R1, a training-free token-level saliency map from logit decomposition plus attention rollout through thinking tokens, and uses saliency-annotation overlap as a GRPO reward to improve faithfulness and interpretability of Qwen2.5-VL reasoning.

Problem

VLMs over-rely on textual cues and produce ungrounded or fabricated reasoning, and their chain-of-thought explanations may not faithfully reflect visual evidence used for answers.

Approach

A cheap saliency map from direct per-token logit contributions (no extra forward/backward passes), extended via attention rollout with thinking tokens as a bottleneck, whose mass inside human bounding boxes is used as a GRPO reward alongside accuracy and format rewards.

Key results

• Saliency-R1-7B reaches 88.1 on POPE (vs 86.7 base) and 88.2 on ChartQA (vs 84.0 base), and 63.7 on SalBench (vs 49.1 base).
• On the saliency-r1-8k test set, Saliency-R1-7B improves Energy-PG from 19.26% to 30.08% and PG from 24.49% to 38.63% over Qwen2.5-VL-7B.
• Saliency map faithfulness on COCO Caption (3B) gives deletion 70.96 at 5% vs 76.42 for the second-best ATTN-LRP.

Surprising findings

• A variant trained with only the saliency reward (no accuracy reward) is only 0.6% lower in average accuracy than the full model.
• SFT on the cold-start data lowered several benchmark scores (e.g., Qwen2.5-VL-7B MMBench 82.8 to 78.8 after SFT).

Limitations

• Bounding boxes are coarse labels; finer segmentation masks may help.
• SFT was run on relatively small data due to compute limits, and some metrics drop after SFT.
• Evaluated only on relatively small base models (3B and 7B).

Experimental setup

Datasets: Vision-R1-cold LLaVA-CoT Mulberry Visual-CoT COCO Caption GranDf OpenPSG MMMU-Pro MMBench POPE MME MME-RealWorld

Benchmarks: MMMU-Pro MMBench POPE MME MME-RealWorld MMStar ChartQA IllusionVQA ScienceQA SalBench COCO Caption GranDf

Metrics: accuracy F1 deletion score insertion score energy-PG PG accuracy normalized log-likelihood

Baselines: CAM Grad-CAM Raw Attention Attention Rollout ATTN-LRP TAM Vision-R1 GPT-4o Claude-3.5 Sonnet Gemini-1.5-Pro InternVL2.5-8B LLaVA-OneVision-7B

Builds on: GRPO Attention Rollout Vision-R1 TAM ATTN-LRP

Closest papers by meaning

• Explainable Saliency: Articulating Reasoning with Contextual Prioritization (CVPR 2025 · similarity 0.90)
• What, Whether and How? Unveiling Process Reward Models for Thinking with Images Reasoning (AAAI 2026 · similarity 0.90)
• On the Faithfulness of Visual Thinking: Measurement and Enhancement (NeurIPS 2026 · similarity 0.90)
• CoFFT: Chain of Foresight-Focus Thought for Visual Language Models (NeurIPS 2025 · similarity 0.90)
• From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models (ICML 2026 · similarity 0.89)
• Through the Lens of Contrast: Self-Improving Visual Reasoning in VLMs (ICLR 2026 · similarity 0.89)