Research map / Video / Video segmentation and tracking
Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Decouples referring video segmentation into static (image-level, via Mask2Former) and motion (temporal, via hierarchical motion perception and contrastive learning) perception, achieving SOTA on five RVOS datasets.
Problem
Referring video segmentation methods reduce the sentence to a single embedding, mixing static image-level cues with temporal motion cues, so motion cues are overshadowed and visually similar objects with different motions are hard to distinguish.
Approach
Split the expression into static cues (nouns, adjectives, prepositions) that drive image-level candidate grounding and motion cues (verbs, adverbs) that drive a motion decoder; add a hierarchical motion perception module that merges object-token trajectories from short-term to long-term; and apply object-wise contrastive learning with a memory bank of target centroids.
Key results
- On MeViS, DsHmp reaches 46.4 J&F, a 9.2-point gain over LMPM (37.2).
- On Ref-YouTube-VOS with Video-Swin-Base, DsHmp reaches 67.1 J&F, above SOC's 66.0.
- On A2D-Sentences with Video-Swin-Base, DsHmp reaches 59.8 mAP versus SgMg's 58.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add sentence decoupling (ablation) | DsHmp · MeViS | J &F: 39.7 -> 42.5 | Helped |
| add hierarchical motion perception (ablation) | DsHmp · MeViS | J &F: 39.7 -> 43.8 | Helped |
| add contrastive learning (ablation) | DsHmp · MeViS | J &F: 39.7 -> 42.1 | Helped |
| remove contrastive learning (ablation) | DsHmp · MeViS | J &F: 46.4 -> 45.1 | Hurt |
| remove hierarchical motion perception (ablation) | DsHmp · MeViS | J &F: 46.4 -> 43.9 | Hurt |
| remove sentence decoupling (ablation) | DsHmp · MeViS | J &F: 46.4 -> 44.9 | Hurt |
| replace decoupled queries with sentence embedding FSe only (ablation) | DsHmp · MeViS | J &F: 46.4 -> 44.9 | Hurt |
| remove sentence embedding FSe from decoupled queries (ablation) | DsHmp · MeViS | J &F: 46.4 -> 45.6 | Hurt |
| bypass learnable queries and directly use static and motion cues (ablation) | DsHmp · MeViS | J &F: 46.4 -> 45.9 | Hurt |
| increase hierarchical stages Nh from 0 to 3 (ablation) | DsHmp · MeViS | J &F: 43.9 -> 46.4 | Helped |
| increase negative samples from 10 to 100 (ablation) | DsHmp · MeViS | J &F: 45.4 -> 46.4 | Helped |
| increase negative samples from 100 to 200 (ablation) | DsHmp · MeViS | J &F: 46.4 -> 46.5 | No clear effect |
Surprising findings
- Using only the basic sentence embedding without decoupled static/motion queries drops MeViS J&F by 1.5 points, and the paper notes that static image-level cues can overshadow motion cues.
- Gains on Ref-YouTube-VOS and Ref-DAVIS17 are modest compared with MeViS, which the authors attribute to sentences that include image-level descriptions and do not strictly require motion.
Limitations
- Gains on Ref-YouTube-VOS and Ref-DAVIS17 are relatively modest because many sentences contain image-level descriptions.
- Contrastive learning for multi-target MeViS samples averages matched video tokens, a heuristic approximation.
- Relies on an external tool to split static and motion words from the sentence.
Experimental setup
Datasets: MeViS Ref-YouTube-VOS Ref-DAVIS17 A2D-Sentences JHMDB-Sentences RefCOCO RefCOCO+ RefCOCOg
Benchmarks: MeViS Ref-YouTube-VOS Ref-DAVIS17 A2D-Sentences JHMDB-Sentences
Metrics: J&F J F mAP oIoU mIoU
Baselines: LMPM ReferFormer SOC SgMg TempCD HTML VLT+TC MTTR
Builds on: Mask2Former ReferFormer LMPM RoBERTa
Closest papers by meaning
- Decoupled Motion Expression Video Segmentation (CVPR 2025 · similarity 0.89)
- Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning (CVPR 2024 · similarity 0.86)
- PixFoundation 2.0: Do Video Multi-Modal LLMs Use Motion in Visual Grounding? (NeurIPS 2026 · similarity 0.86)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.85)
- FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understanding (NeurIPS 2025 · similarity 0.85)
- Motion-Grounded Video Reasoning: Understanding and Perceiving Motion at Pixel Level (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.