Research map / Vision-language and multimodal / Human and embodied modeling
Language-Guided Salient Object Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LG-SOR, which uses LVLM-generated scene descriptions (entity and relation cues parsed with spaCy) to guide salient object ranking via a text-guided visual modulation module and a text-aware graph reasoning module.
Problem
Salient object ranking predicts the order in which humans shift attention across objects, but existing methods overlook or only implicitly model semantic relations among salient objects, producing rankings inconsistent with humans.
Approach
LVLM descriptions tend to start with the most salient object and describe relations in order, so their text can supply semantic relation cues and implicit entity order. TGVM modulates visual features with word and sentence text features to suppress background noise, and TAVR builds a multimodal graph whose edges are weighted by parsed relation cues.
Key results
- On ASSR, LG-SOR (Swin-L) reaches SA-SOR 0.787, SOR 0.895, MAE 0.049, versus PoseSOR's 0.664 SA-SOR and DSGNN's 0.765 SA-SOR.
- On IRSR, LG-SOR (Swin-L) achieves SA-SOR 0.634, SOR 0.835, MAE 0.050, the best across all compared methods.
- Ablation: adding TGVM and TAVR raises ASSR SA-SOR from 0.687 (baseline) to 0.733 with ResNet-50.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use LG-SOR with ResNet-50 instead of the strongest reported baseline | LG-SOR · ASSR Test Set | SA-SOR: 0.772 -> 0.733 | Mixed |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · ASSR Test Set | SA-SOR: 0.772 -> 0.787 | Helped |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · ASSR Test Set | SOR: 0.875 -> 0.895 | Helped |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · ASSR Test Set | MAE: 0.052 -> 0.049 | Helped |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · IRSR Test Set | SA-SOR: 0.618 -> 0.634 | Helped |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · IRSR Test Set | SOR: 0.825 -> 0.835 | Helped |
| use LG-SOR with Swin-L instead of the strongest reported baseline | LG-SOR · IRSR Test Set | MAE: 0.049 -> 0.050 | No clear effect |
| add word-feature cross-attention (ablation) | LG-SOR baseline on ASSR · ASSR | SA-SOR: 0.687 -> 0.704 | Helped |
| add sentence-feature regression block (ablation) | TGVM on ASSR, after word-feature cross-attention · ASSR | SA-SOR: 0.704 -> 0.706 | Helped |
| add gate block (ablation) | TGVM on ASSR, after word- and sentence-level features · ASSR | SA-SOR: 0.706 -> 0.713 | Helped |
| add graph reasoning using entity-aligned object embeddings and relation cues (ablation) | LG-SOR on ASSR · ASSR | SA-SOR: 0.713 -> 0.733 | Helped |
| shuffle the parsed entity order instead of preserving its normal order (ablation) | LG-SOR on ASSR · ASSR | SA-SOR: 0.733 -> 0.726 | Hurt |
Surprising findings
- GPT-4V captions give higher scores than InstructBLIP captions, but InstructBLIP was chosen for accessibility and cost.
- Short InstructBLIP descriptions cause a clear performance drop relative to long descriptions.
Limitations
- Struggles when salient objects share similar positions and attributes and the description lacks semantic relations.
- Depends on LVLM-generated descriptions whose quality affects results.
Experimental setup
Datasets: ASSR IRSR MS-COCO
Benchmarks: ASSR IRSR
Metrics: SA-SOR SOR MAE
Baselines: S4Net VST QueryInst Mask2Former GiT X-Decoder RSDNet ASSR IRSR PPA PSR HyperSOR
Builds on: Mask2Former IRSR X-Decoder BERT graph convolution
Closest papers by meaning
- SeqRank: Sequential Ranking of Salient Objects (AAAI 2024 · similarity 0.91)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.89)
- Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward (CVPR 2026 · similarity 0.87)
- Explainable Saliency: Articulating Reasoning with Contextual Prioritization (CVPR 2025 · similarity 0.87)
- Seeing What Matters: A Training-Free Self-Guided Framework for Multimodal Detail Perception and Reasoning (CVPR 2026 · similarity 0.87)
- What's in the Image? A Deep-Dive into the Vision of Vision Language Models (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.