Research map / Video / Video segmentation and tracking
A Motion-aware Spatio-temporal Graph for Video Salient Object Ranking
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a unified spatial-temporal graph with motion-aware instance-level temporal contrast for video salient object ranking, plus a saliency-ranking-based video retargeting method.
Problem
Video salient object ranking requires modeling how the saliency order of objects changes over time, but prior methods use only global inter-frame features and stage-wise spatial-then-temporal fusion, ignoring instance-level motion.
Approach
Approximates instance motion by comparing an instance's features with features from the same doubled-size region in adjacent frames, and jointly reasons over multi-scale spatial contrasts and intra/inter-instance temporal correlations in a single graph instead of stage-wise fusion.
Key results
- On RVSOD the method reaches SA-SOR 0.603 and MAE 0.0698, versus 0.563 SA-SOR for the Liu graph baseline and 0.560 for Lin VSOR.
- On DAVSOD it reaches SA-SOR 0.207 and MAE 0.0626, versus 0.179 SA-SOR for the Liu baseline.
- Joint spatial-temporal graph reasoning outperforms spatial-then-temporal (0.571) and temporal-then-spatial (0.590) fusion on RVSOD SA-SOR.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add global-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · RVSOD | SA-SOR: 0.563 -> 0.585 | Helped |
| add global-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · DAVSOD | SA-SOR: 0.179 -> 0.186 | Helped |
| add instance-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · RVSOD | SA-SOR: 0.563 -> 0.591 | Helped |
| add instance-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · DAVSOD | SA-SOR: 0.179 -> 0.194 | Helped |
| add motion-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · RVSOD | SA-SOR: 0.563 -> 0.587 | Helped |
| add motion-aware temporal relation modeling (ablation) | Liu baseline without temporal relation modeling · DAVSOD | SA-SOR: 0.179 -> 0.195 | Helped |
| combine GTRM, ITRM, and MTRM jointly (ablation) | Liu baseline without temporal relation modeling · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.563 -> 0.603; DAVSOD: 0.179 -> 0.207 | Helped |
| increase motion-comparison bounding-box expansion from 1× to 2× (ablation) | Motion-aware temporal relation modeling · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.6092 -> 0.629; DAVSOD: 0.2421 -> 0.246 | Helped |
| increase motion-comparison bounding-box expansion from 2× to 3× (ablation) | Motion-aware temporal relation modeling · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.629 -> 0.5900; DAVSOD: 0.246 -> 0.2374 | Hurt |
| use joint spatial-temporal graph reasoning instead of spatial-then-temporal fusion (ablation) | Liu baseline without temporal relation modeling · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.571 -> 0.603; DAVSOD: 0.184 -> 0.207 | Helped |
| use joint spatial-temporal graph reasoning instead of temporal-then-spatial fusion (ablation) | Liu baseline without temporal relation modeling · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.590 -> 0.603; DAVSOD: 0.183 -> 0.207 | Helped |
| use the proposed VSOR model instead of Liu | Video salient object ranking · RVSOD and DAVSOD | SA-SOR: RVSOD: 0.563 -> 0.603; DAVSOD: 0.179 -> 0.207 | Helped |
Surprising findings
- Performance on DAVSOD is low overall (SA-SOR 0.207), attributed partly to label noise from varying fixations across frames and occlusion.
- Expanding the motion-comparison box beyond 2x degrades results, as extra background noise is introduced.
Limitations
- Relies on an object detector that is not saliency-sensitive, so non-salient objects may still be detected.
- Results on DAVSOD are poor due to occlusion, small objects and inconsistent fixation-based labels.
- Lin's method was not open-source, so comparison on DAVSOD is missing for that baseline.
Experimental setup
Datasets: RVSOD DAVSOD MS-COCO 2017
Benchmarks: RVSOD DAVSOD
Metrics: SA-SOR MAE
Baselines: Fang PPA Liu SA-SOR graph model PSR SeqRank DCFNet SCOTCH Lin VSOR
Builds on: Liu graph-based salient object ranking Mask R-CNN PPA smartVidCrop
Closest papers by meaning
- Revisiting Video Saliency: A Large-Scale Benchmark and a New Model (CVPR 2018 · similarity 0.91)
- Going From Image to Video Saliency: Augmenting Image Salience With Dynamic Attentional Push (CVPR 2018 · similarity 0.89)
- CaRDiff: Video Salient Object Ranking Chain of Thought Reasoning for Saliency Prediction with Diffusion (AAAI 2025 · similarity 0.88)
- Probabilistic Salient Object Ranking (ICML 2026 · similarity 0.88)
- Language-Guided Salient Object Ranking (CVPR 2025 · similarity 0.87)
- Advancing Saliency Ranking with Human Fixations: Dataset Models and Benchmarks (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.