Research map / Video / Video segmentation and tracking
Learning Spatial-Semantic Features for Robust Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes S3, a video object segmentation framework that fuses ViT CLS-token semantics with CNN multi-scale features via spatial-semantic blocks and propagates discriminative object queries, reaching SOTA on DAVIS, YouTube-VOS, MOSE and LVOS.
Problem
Video object segmentation of multiple similar objects with complex or separate parts in long-term videos, where pixel-wise correlation lacks instance-level semantics and query propagation accumulates noise.
Approach
A spatial-semantic block injects global ViT CLS-token semantics into multi-scale CNN features via cross-attention and then models spatial dependencies with multi-scale deformable attention, combined with discriminative query propagation that updates object queries using the most target-similar channel activations.
Key results
- Trained on MEGA, S3 reaches 87.8% J&F on DAVIS 2017 test and 88.1% G on YouTube-VOS 2019 at 480p input.
- On MOSE val S3 reaches 74.0% J&F and on LVOS test 73.0%, versus 66.7% for Cutie on LVOS test with MEGA.
- Trained on YouTubeVOS and DAVIS only, S3 reaches 68.5% J&F on MOSE val, and 66.5% on LVOS test, versus 64.0% and 56.2% for Cutie.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add discriminative query propagation (ablation) | Cutie with ResNet features · DAVIS 2017 test | J&F: 84.2 -> 85.2 | Helped |
| Add discriminative query propagation (ablation) | Cutie with ResNet features · LVOS test | J&F: 56.2 -> 57.4 | Helped |
| Replace ResNet features with ViT features (ablation) | Cutie + discriminative query · MOSE-val | J&F: 64.2 -> 64.2 | No clear effect |
| Add spatial-semantic blocks (ablation) | Cutie + discriminative query + ViT · MOSE-val | J&F: 64.2 -> 68.5 | Helped |
| Add spatial-semantic blocks (ablation) | Cutie + discriminative query + ViT · LVOS test | J&F: 58.3 -> 66.5 | Helped |
| Use MAE pre-trained weights instead of ResNet weights (ablation) | ResNet + discriminative query · MOSE-val | J&F: 64.2 -> 67.0 | Helped |
| Use DINOv2 pre-trained weights instead of ResNet weights (ablation) | ResNet + discriminative query · MOSE-val | J&F: 64.2 -> 67.5 | Helped |
| Use ViT-small pre-trained weights instead of ResNet weights (ablation) | ResNet + discriminative query · MOSE-val | J&F: 64.2 -> 64.3 | No clear effect |
| Use Depth Anything pre-trained weights instead of ResNet weights (ablation) | ResNet + discriminative query · MOSE-val | J&F: 64.2 -> 68.5 | Helped |
| Increase query count from 8 to 16 (ablation) | S3 · MOSE-val | J&F: 68.5 -> 67.7 | Hurt |
| Increase query count from 8 to 32 (ablation) | S3 · MOSE-val | J&F: 68.5 -> 68.3 | Hurt |
| Replace spatial-semantic fusion with Adapter (ablation) | S3 trained on MEGA · MOSE-val | J&F: 74.0 -> 72.2 | Hurt |
Surprising findings
- Directly adding a ViT (ViTDet-style multi-scale features) gave only minimal gains over ResNet, while the spatial-semantic design gave larger improvements.
- Depth Anything pre-trained weights gave notably better results than MAE, DINOv2, or ViT-small backbones.
Limitations
- Tends to segment the whole object instead of a specified part, as shown in VOT part-tracking failure cases.
- Inference speed is lower than some baselines (13.1 FPS vs 36.4 FPS for Cutie), and the authors defer speed improvements to future work.
Experimental setup
Datasets: YouTubeVOS DAVIS 2017 OVIS MOSE BURST MEGA
Benchmarks: DAVIS 2017 test YouTube-VOS 2018 val YouTube-VOS 2019 val MOSE val LVOS test LVOS val LVOS v2 val VOTS2023
Metrics: J&F J F G
Baselines: Cutie XMem DEVA JointFormer ISVOS SimVOS SAM 2 DeAOT STCN MiVOS
Builds on: Cutie XMem ViT-Adapter ViT-CoMer Mask2Former
Closest papers by meaning
- Feature Space Optimization for Semantic Video Segmentation (CVPR 2016 · similarity 0.89)
- Target-Aware Object Discovery and Association for Unsupervised Video Multi-Object Segmentation (CVPR 2021 · similarity 0.89)
- SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation (NeurIPS 2023 · similarity 0.88)
- Segment Any Motion in Videos (CVPR 2025 · similarity 0.87)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.87)
- Exploiting Temporal State Space Sharing for Video Semantic Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.