Research map / Video / Video segmentation and tracking
Decoupled Motion Expression Video Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DMVS, which freezes a VITA video instance segmenter and trains a lightweight motion-expression module (static/motion cue encoder plus video-query-initialized motion query decoder) for motion expression video segmentation on MeViS.
Problem
Motion expression video segmentation (MEVS) on MeViS requires segmenting multiple objects from motion descriptions, but prior methods inject text into a VIS model and retrain it end-to-end, which is hard to optimize and costly.
Approach
Decouple MEVS into a frozen query-based VIS model (VITA) producing frame and video queries, plus a trainable motion expression module that splits expressions into static and motion cues interacting with frame and video queries respectively, and initializes motion queries from Top-K classification-ranked video queries.
Key results
- DMVS achieves 48.6 J&F on MeViS val, outperforming DsHmp (46.4) by 2.2 points.
- On MeViS valu, DMVS reaches 58.3 J&F versus DsHmp's 55.3.
- DMVS has 12.7M learnable parameters vs 102.3M for DsHmp and trains in 7 hours vs 17 hours.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DMVS instead of DsHmp | MeViS · MeViS val | J&F: 46.4 -> 48.6 | Helped |
| Use DMVS instead of DsHmp | MeViS · MeViS valu | J&F: 55.3 -> 58.3 | Helped |
| Use DMVS instead of DsHmp | RVOS models · Ref-YouTube-VOS | J&F: 63.6 -> 64.3 | Helped |
| Use DMVS instead of DsHmp | RVOS models · Ref-DAVIS17 | J&F: 64.0 -> 65.2 | Helped |
| Add motion query initialization from classification-ranked video queries (ablation) | DMVS baseline on MeViS · MeViS val | J&F: 44.2 -> 46.3 | Helped |
| Add the motion expression encoder (ablation) | DMVS with MQI on MeViS · MeViS val | J&F: 46.3 -> 48.6 | Helped |
| Add sentence embedding interaction with frame and video queries (ablation) | Motion expression encoder · MeViS val | J&F: 46.3 -> 47.1 | Helped |
| Add static cues to frame queries (ablation) | Motion expression encoder · MeViS val | J&F: 47.1 -> 47.6 | Helped |
| Add motion cues to video queries (ablation) | Motion expression encoder · MeViS val | J&F: 47.1 -> 47.8 | Helped |
| Initialize motion queries from classification-ranked video queries instead of random initialization (ablation) | DMVS · MeViS val | J&F: 46.5 -> 48.6 | Helped |
| Set motion query count Nm to 10 instead of 20 (ablation) | DMVS · MeViS val | J&F: 48.6 -> 47.8 | Hurt |
| Set motion query count Nm to 50 or 100 instead of 20 (ablation) | DMVS · MeViS val | J&F: 48.6 -> 47.3; 48.6 -> 46.2 | Hurt |
Surprising findings
- Ref-YouTube-VOS gains are small (0.7 J&F), attributed to its static expressions with little motion dependence.
Limitations
- Gains on Ref-YouTube-VOS are marginal because that dataset emphasizes static expressions.
- Performance depends on the pretrained VIS weights; OVIS pretraining was best among those tested.
Experimental setup
Datasets: MeViS Ref-YouTube-VOS Ref-COCO Ref-COCO+ Ref-COCOg OVIS
Benchmarks: MeViS Ref-YouTube-VOS Ref-DAVIS17
Metrics: J&F J F
Baselines: LMPM DsHmp URVOS LBDT MTTR ReferFormer VLT+TC OnlineRefer HTML R2VOS SgMg TempCD
Builds on: VITA Mask2Former LMPM DsHmp ReferFormer
Closest papers by meaning
- Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation (CVPR 2024 · similarity 0.89)
- Efficient Video Instance Segmentation via Tracklet Query and Proposal (CVPR 2022 · similarity 0.89)
- End-to-End Video Instance Segmentation With Transformers (CVPR 2021 · similarity 0.88)
- Segment Any Motion in Videos (CVPR 2025 · similarity 0.87)
- SyncVIS: Synchronized Video Instance Segmentation (NeurIPS 2024 · similarity 0.86)
- Learning Segmentation from Point Trajectories (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.