Research map / Video / Video segmentation and tracking
Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation
Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, Gang Hua, Zixin Zhu
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add referring text tokens as condition instead of image tokens only (ablation) | VD-IT · Ref-YouTube-VOS | J&F: 59.7 -> 61.9 | Improved |
| combine image and text tokens as condition (ablation) | VD-IT · Ref-YouTube-VOS | J&F: 61.9 -> 63.8 | Improved |
| predict video-specific noise instead of Gaussian noise (ablation) | VD-IT · Ref-YouTube-VOS | J&F: 63.8 -> 64.8 | Improved |
| fuse text and image tokens by attention instead of concatenation (ablation) | VD-IT · Ref-YouTube-VOS | J&F: 62.4 -> 64.8 | Improved |
| increase noise step added to video latent from 1 to 100 (ablation) | VD-IT · Ref-YouTube-VOS | J&F: 64.8 -> 62.7 | Hurt |
| use fixed generative T2V diffusion backbone instead of Video Swin | R-VOS · Ref-YouTube-VOS | IoU diff between frames: 7.24 -> 5.19 | Improved |
| add text tokens to image-token conditioning (ablation) | VD-IT · Ref-YouTube-VOS | ratio of IoU>0.9 to IoU>0.5 masks: 47.26% -> 46.71% | Mixed |
| apply VD-IT unchanged to referring image segmentation | VD-IT · RefCOCO/+/g | Overall IoU: 76.3 -> 76.7 (RefCOCO) | No clear effect |
| use T2V diffusion backbone instead of Video Swin at inference | R-VOS · Ref-YouTube-VOS | FPS: 65 -> 21 | Hurt |
Closest papers by meaning
- ReferEverything: Towards Segmenting Everything We Can Speak of in Videos (ICCV 2025 · similarity 0.92)
- VidSeg: Training-free Video Semantic Segmentation based on Diffusion Models (CVPR 2025 · similarity 0.91)
- Deforming Videos to Masks: Flow Matching for Referring Video Segmentation (ICLR 2026 · similarity 0.91)
- End-to-End Referring Video Object Segmentation With Multimodal Transformers (CVPR 2022 · similarity 0.90)
- LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation (CVPR 2024 · similarity 0.89)
- Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
