Research map / Video / Video segmentation and tracking
Semantic and Sequential Alignment for Referring Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SSA, a CLIP-based RVOS framework with a lightweight semantic alignment adapter and a sequential object alignment module (Hungarian-matched trajectories, temporal enhancement, and trajectory ensembling) that outperforms prior methods on MeViS, Ref-YouTube-VOS and Ref-DAVIS17.
Problem
Referring video object segmentation struggles with the semantic gap between linguistic descriptions and video objects, and with inconsistent per-frame instance queries under motion, occlusion and multi-target expressions.
Approach
Use a frozen CLIP (ConvNeXt) as a joint visual-language encoder with a multi-granularity cross-attention adapter for semantic alignment, and align instance queries across time by Hungarian-matched trajectories with temporal enhancement and attention-weighted trajectory ensembling into a global video query.
Key results
- Achieves 48.6 J&F on MeViS Val, surpassing DsHmp (46.4) by 2.2 points and VISA-7B (43.5) with fewer trainable parameters (~46.5M).
- Reaches 64.3 J&F on Ref-Youtube-VOS and 67.3 J&F on Ref-DAVIS17, improving over DsHmp by 0.7 and over 3.3 points respectively.
- Ablation on MeViS shows semantic alignment adds 3.9 J&F and sequential alignment adds 2.7 J&F over the baseline; full model gains 6.2 J&F.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic feature alignment (ablation) | SSA baseline · MeViS Val | J&F: 42.4 -> 46.3 | Helped |
| add sequential object alignment (ablation) | SSA baseline · MeViS Val | J&F: 42.4 -> 45.1 | Helped |
| add semantic feature alignment and sequential object alignment (ablation) | SSA baseline · MeViS Val | J&F: 42.4 -> 48.6 | Helped |
| replace DsHmp encoders with CLIP encoders (ablation) | DsHmp · MeViS Val | J&F: 46.4 -> 47.2 | Helped |
| add instance correspondence (ablation) | DsHmp with CLIP encoders · MeViS Val | J&F: 47.2 -> 47.6 | Helped |
| use frozen CLIP instead of standalone encoders (ablation) | SSA baseline · MeViS Val | J&F: 45.1 -> 47.5 | Helped |
| fine-tune the last CLIP stage instead of using frozen CLIP alone (ablation) | SSA · MeViS Val | J&F: 47.5 -> 47.9 | Helped |
| add the proposed adapter to frozen CLIP (ablation) | SSA · MeViS Val | J&F: 47.5 -> 48.6 | Helped |
| remove bidirectional attention (ablation) | SSA sequential object alignment · MeViS Val | J&F: 48.6 -> 47.8 | Hurt |
| remove trajectory ensemble (ablation) | SSA sequential object alignment · MeViS Val | J&F: 48.6 -> 47.2 | Hurt |
| remove Hungarian trajectory generation, temporal enhancement, and trajectory ensemble (ablation) | SSA sequential object alignment · MeViS Val | J&F: 48.6 -> 46.1 | Hurt |
Surprising findings
- Frozen CLIP with the adapter outperforms fine-tuning the last CLIP stage while using fewer trainable parameters (46.5M vs 100.6M).
- Adding the DsHmp correspondence component alone only partly explains the gain; CLIP encoders alone raise DsHmp by just 0.8 J&F on MeViS.
Limitations
- Gains on the Ref-YouTube-VOS dataset are modest (0.7 J&F over DsHmp).
- Compute hardware and training time are not reported.
Experimental setup
Datasets: MeViS Ref-Youtube-VOS RefCOCO RefCOCO+ RefCOCOg
Benchmarks: MeViS Ref-Youtube-VOS Ref-DAVIS17
Metrics: J&F J F
Baselines: LMPM DsHmp ReferFormer MTTR HTR VISA-7B VideoLISA-3.8B Grounded-SAM 2 AL-Ref-SAM 2 OnlineRefer
Builds on: CLIP Mask2Former DsHmp LMPM ReferFormer
Closest papers by meaning
- SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation (NeurIPS 2023 · similarity 0.94)
- Multi-Level Representation Learning With Semantic Alignment for Referring Video Object Segmentation (CVPR 2022 · similarity 0.92)
- Deforming Videos to Masks: Flow Matching for Referring Video Segmentation (ICLR 2026 · similarity 0.92)
- LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation (CVPR 2024 · similarity 0.91)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.89)
- Exploring Temporal Feature Correlation for Efficient and Stable Video Semantic Segmentation (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.