Research map / Video / Video-language understanding
CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CaRDiff, which uses an MLLM with a chain-of-thought prompt (VSOR-CoT) to caption videos and rank salient objects, then feeds grounded ranking maps plus video features into a diffusion model to predict saliency maps.
Problem
Video saliency prediction typically models perceptual features and ignores the language-driven reasoning (object ranking) that guides human gaze.
Approach
VSOR-CoT prompts an MLLM to caption the video and reason out a ranked list of salient objects; a grounding module (GroundingDINO) localizes them, producing grayscale ranking maps that condition a diffusion decoder for saliency prediction.
Key results
- On MVS, CaRDiff reaches AUC-J 0.870, CC 0.714, SIM 0.630, NSS 1.685, the best among compared methods (MVFormer: CC 0.687, NSS 1.646).
- Zero-shot transfer trained on MVS and tested on DHF1k gives CC 0.312 and NSS 1.584, above Diff-Sal and MVFormer.
- Ablation: fine-tuned VSOR-CoT improves CC from 0.700 to 0.714 over the variant without it; applying ranking maps to 1/4 of input frames gives the best result.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VSOR-CoT during fine-tuning (ablation) | CaRDiff · MVS | CC: 0.700 -> 0.714 | Helped |
| Add VSOR-CoT in the zero-shot setting (ablation) | CaRDiff · MVS | CC: 0.626 -> 0.659 | Helped |
| Use fine-tuned VSOR-CoT instead of zero-shot VSOR-CoT (ablation) | CaRDiff · MVS | CC: 0.659 -> 0.714 | Helped |
| Set ranking map ratio to 1/4 rather than 0 (ablation) | CaRDiff · MVS | CC: 0.700 -> 0.714 | Helped |
| Set ranking map ratio to 1/16 rather than 1/4 (ablation) | CaRDiff · MVS | CC: 0.714 -> 0.709 | Hurt |
| Set ranking map ratio to 1/8 rather than 1/4 (ablation) | CaRDiff · MVS | CC: 0.714 -> 0.712 | Hurt |
| Set ranking map ratio to 1/2 rather than 1/4 (ablation) | CaRDiff · MVS | CC: 0.714 -> 0.710 | Hurt |
| Set ranking map ratio to 1 rather than 1/4 (ablation) | CaRDiff · MVS | CC: 0.714 -> 0.704 | Hurt |
| Replace predicted ranking maps with random ranking maps (ablation) | CaRDiff · MVS | accuracy | Hurt |
| Use CaRDiff instead of MVFormer | video saliency prediction · MVS | CC: 0.687 -> 0.714 | Helped |
| Use CaRDiff instead of MVFormer | zero-shot video saliency prediction · DHF1k validation | NSS: 1.501 -> 1.584 | Helped |
Surprising findings
- Replacing predicted ranking maps with random ones degrades accuracy, showing the ranking guidance matters.
Limitations
- Performance depends on the accuracy of the grounding module.
- Combining an MLLM and a diffusion model increases computational cost, limiting real-time or resource-constrained use.
Experimental setup
Datasets: MVS LCS-558K DHF1k K400
Benchmarks: MVS DHF1k
Metrics: AUC-J CC SIM NSS
Baselines: ITTI GBVS SALICON SalGAN SAM TASED ViNet HD2S UNISAL Diff-Sal MVFormer VSFT
Builds on: LLaVA GroundingDINO Diff-Sal MViT-v2 VILA-1.5
Closest papers by meaning
- Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (CVPR 2018 · similarity 0.90)
- Language-Guided Salient Object Ranking (CVPR 2025 · similarity 0.89)
- A Motion-aware Spatio-temporal Graph for Video Salient Object Ranking (NeurIPS 2024 · similarity 0.88)
- Explainable Saliency: Articulating Reasoning with Contextual Prioritization (CVPR 2025 · similarity 0.87)
- Saliency-R1: Enforcing Interpretable and Faithful Vision-language Reasoning via Saliency-map Alignment Reward (CVPR 2026 · similarity 0.87)
- DiffSal: Joint Audio and Video Learning for Diffusion Saliency Prediction (CVPR 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.