Research map / Vision-language and multimodal / Human and embodied modeling
Explainable Saliency: Articulating Reasoning with Contextual Prioritization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes XSal, an explainable saliency model that uses a VLM (MiniCPM-V 2.6) to generate reasoning steps and semantic proposals, grounds them with ClipSeg into semantic prototypes, and selects the top-K prototypes via contextual prioritization to predict saliency.
Problem
Deep saliency models are black boxes that do not explain why regions are predicted as salient, limiting trust in settings where the rationale matters.
Approach
Combine VLM-generated step-by-step reasoning and semantic proposals (grounded via open-vocabulary segmentation into prototypes) with a hard top-K contextual prioritization over prototypes, plus a progressive K reduction schedule, so saliency is a weighted sum of a few explained semantic elements.
Key results
- On AiR, XSal reaches NSS 1.94 and CC 0.70, versus 1.85/0.66 for GazeXplain and 1.76/0.63 for Chen et al.
- On OSIE, XSal reaches NSS 2.96, the highest, though CC 0.72 trails GazeXplain's 0.74.
- On AiR explainability, XSal lowers AUC-E from 0.899 to 0.713 and raises AOPC from 0.241 to 0.748 relative to Chen et al.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add VLM reasoning (ablation) | XSal baseline · AiR | NSS: 1.777 -> 1.917 | Helped |
| add VLM reasoning (ablation) | XSal baseline · OSIE | NSS: 2.835 -> 2.904 | Helped |
| add VLM reasoning (ablation) | XSal baseline · AiR | AUC-E: 0.910 -> 0.700 | Helped |
| add VLM reasoning (ablation) | XSal baseline · OSIE | AOPC: 0.567 -> 0.876 | Helped |
| add contextual prioritization to the reasoning model (ablation) | XSal · AiR | NSS: 1.917 -> 1.942 | Helped |
| add contextual prioritization to the reasoning model (ablation) | XSal · OSIE | NSS: 2.904 -> 2.965 | Helped |
| add contextual prioritization to the reasoning model (ablation) | XSal · AiR | AUC-E: 0.700 -> 0.713 | Hurt |
| add contextual prioritization to the reasoning model (ablation) | XSal · OSIE | AOPC: 0.876 -> 1.115 | Helped |
| use XSal instead of GazeXplain | saliency prediction · OSIE | CC: 0.74 -> 0.72 | Hurt |
Surprising findings
- Saliency CC is lower than GazeXplain on OSIE, which the authors attribute to discarding low-agreement regions via prioritization.
Limitations
- Lower CC on OSIE than GazeXplain, since low-saliency regions are discarded by prioritization
- Explanations depend on the VLM's reasoning and on open-vocabulary segmentation quality
Experimental setup
Datasets: AiR OSIE
Benchmarks: AiR OSIE
Metrics: NSS CC AUC AUC-E AOPC LOdds
Baselines: SAM SALICON DINet Chen et al. GazeXplain
Builds on: DINet Chen et al. explainable saliency GazeXplain CLIPSeg
Closest papers by meaning
- Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward (CVPR 2026 · similarity 0.90)
- Rethinking Saliency Maps: A Cognitive Human Aligned Taxonomy and Evaluation Framework for Explanations (AAAI 2026 · similarity 0.90)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.87)
- Language-Guided Salient Object Ranking (CVPR 2025 · similarity 0.87)
- Revealing the Gap in Human and VLM Scene Perception through Counterfactual Semantic Saliency (NeurIPS 2026 · similarity 0.86)
- Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.