Research map / Video / Video segmentation and tracking
LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes LoSh, which pairs each long referring expression with a subject-centric short version and jointly predicts masks for both, adding a long-short cross-attention, a long-short predictions intersection loss, and a forward-backward optical-flow visual consistency loss on top of MTTR and SgMg.
Problem
Referring video object segmentation models over-attend to action- and relation-related words in long expressions, leading to partial or wrong instance masks instead of focusing on appearance.
Approach
Derive a subject-centric short expression (words before the first verb) and jointly predict masks for long and short expressions; a unidirectional cross-attention from short to long features and a loss requiring the long-expression mask to lie within the short-expression mask regularize the model toward appearance. Also warps neighboring-frame features to annotated frames using Farneback optical f
Key results
- On A2D-Sentences, LoSh-M (Video-Swin-T) improves over MTTR by +3.1 mAP, +2.7% Overall IoU and +3.1% Mean IoU (47.8 mAP, 72.9 Overall IoU).
- LoSh-S with Video-Swin-B reaches 59.9 mAP / 81.2 Overall IoU on A2D-Sentences, above SgMg's 58.5 / 79.9.
- On Refer-YouTube-VOS with Video-Swin-T, LoSh-S improves SgMg by +1.7 J&F (63.7 vs 62.0).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LoSh joint prediction components to MTTR | MTTR (Video-Swin-T) · A2D-Sentences | mAP: 44.7 -> 47.8 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-T) · A2D-Sentences | mAP: 56.1 -> 57.6 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-B) · A2D-Sentences | mAP: 58.5 -> 59.9 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-T) · Refer-YouTube-VOS | J &F: 62.0 -> 63.7 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-B) · Refer-YouTube-VOS | J &F: 65.7 -> 67.2 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-T) · Refer-DAVIS17 | J &F: 61.9 -> 62.9 | Helped |
| Add LoSh joint prediction components to SgMg | SgMg (Video-Swin-B) · Refer-DAVIS17 | J &F: 63.3 -> 64.3 | Helped |
| Add short-text prediction branch (ablation) | LoSh-M · A2D-Sentences | mAP: 45.4 -> 47.8 | Helped |
| Add long-short predictions intersection loss (ablation) | LoSh-M · A2D-Sentences | mAP: 46.1 -> 47.8 | Helped |
| Add long-short cross-attention (ablation) | LoSh-M · A2D-Sentences | mAP: 47.3 -> 47.8 | Helped |
| Use inverse cross-attention instead of the proposed direction (ablation) | LoSh-M · A2D-Sentences | mAP: 47.8 -> 47.2 | Hurt |
| Use bidirectional long-short cross-attention (ablation) | LoSh-M · A2D-Sentences | mAP: 47.8 -> 47.6 | Hurt |
Surprising findings
- Reversing the cross-attention direction (Fl to Fs) performed worse than no cross-attention at all.
- Reverse-direction or mutual flow-warped consistency losses gave no gain over the default forward-backward loss.
Limitations
- Short expressions are extracted heuristically (words before first verb via NLTK tagger), and around 10% of short expressions map to multiple instances requiring merged masks.
- Optical flow is computed offline with Farneback rather than learned, adding CPU preprocessing time during training.
Experimental setup
Datasets: A2D-Sentences Refer-YouTube-VOS JHMDB-Sentences Refer-DAVIS17 RIS datasets
Benchmarks: A2D-Sentences Refer-YouTube-VOS JHMDB-Sentences Refer-DAVIS17
Metrics: Overall IoU Mean IoU mAP J&F J F
Baselines: MTTR SgMg ReferFormer CMPC-V YOFO MLRL Gavrilyuk et al.
Builds on: MTTR SgMg ReferFormer Video Swin Transformer RoBERTa
Closest papers by meaning
- Semantic and Sequential Alignment for Referring Video Object Segmentation (CVPR 2025 · similarity 0.91)
- SAMWISE: Infusing Wisdom in SAM2 for Text-Driven Video Segmentation (CVPR 2025 · similarity 0.90)
- ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentation (CVPR 2025 · similarity 0.87)
- Boosting Weakly Supervised Referring Image Segmentation via Progressive Comprehension (NeurIPS 2024 · similarity 0.87)
- DTOS: Dynamic Time Object Sensing with Large Multimodal Model (CVPR 2025 · similarity 0.87)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.