Research map / Video / Video-language understanding
Learning to Segment Referred Objects from Narrated Egocentric Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ROSA, a weakly-supervised framework that aligns narration noun phrases with SAM mask proposals using CLIP-based dual encoders and global-local contrastive learning, and introduces the VISOR-NVOS benchmark.
Problem
Pixel-level grounding of objects referred to in narrations of egocentric videos, without any spatial (mask or box) annotations for referred objects during training.
Approach
Combines a video-narration contrastive loss, using a temporally adaptive pooling of region-phrase similarities as a MIL-style affinity, with a region-phrase contrastive loss on pseudo-labelled alignments; region embeddings come from a mask-guided CLIP image encoder that shares computation across SAM mask proposals.
Key results
- ROSA (ViT-L/14) reaches 42.4 J&F on VISOR-NVOS test, versus 39.5 for GroundedSAM (trained with boxes) and 24.0 for SAM+CLIP.
- ROSA (ViT-B/16) improves J&F by 14.1% over the SAM+CLIP baseline on VISOR-NVOS with the same backbone.
- On YouCook2-BB zero-shot, ROSA gets 69.35% point accuracy vs 59.25% for CoMMa, and 37.93 box accuracy, comparable to models trained on YouCook2.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train ROSA with weakly supervised video-narration and region-phrase alignment | SAM + CLIP · VISOR-NVOS test | J &F: 24.0 -> 38.1 | Helped |
| Use CLIP ViT-L/14 instead of ViT-B/16 | ROSA · VISOR-NVOS test | J &F: 38.1 -> 42.4 | Helped |
| Add video-narration contrastive training (VNA) (ablation) | CLIP-based Dual Encoder · VISOR-NVOS test | J &F: 28.1 -> 36.0 | Helped |
| Add region-to-phrase loss (R2P) to VNA (ablation) | ROSA training objective · VISOR-NVOS test | J &F: 36.0 -> 37.0 | Helped |
| Add phrase-to-region loss (P2R) to VNA (ablation) | ROSA training objective · VISOR-NVOS test | J &F: 36.0 -> 37.3 | Helped |
| Add both region-to-phrase and phrase-to-region losses (ablation) | ROSA training objective · VISOR-NVOS test | J &F: 36.0 -> 38.1 | Helped |
| Use negative regions aligned with other phrases in the batch (ablation) | ROSA region-phrase contrastive learning · VISOR-NVOS test | J &F: 31.3 -> 38.1 | Helped |
| Use all other regions in the batch as negative samples instead of aligned regions (ablation) | ROSA region-phrase contrastive learning · VISOR-NVOS test | J &F: 38.1 -> 31.1 | Hurt |
| Use regions from the same video as negative samples instead of aligned regions (ablation) | ROSA region-phrase contrastive learning · VISOR-NVOS test | J &F: 38.1 -> 31.3 | Hurt |
| Use regions from other videos as negative samples instead of aligned regions (ablation) | ROSA region-phrase contrastive learning · VISOR-NVOS test | J &F: 38.1 -> 30.3 | Hurt |
| Use adaptive temporal aggregation instead of random-frame selection (ablation) | ROSA temporal aggregation · VISOR-NVOS test | J &F: 36.6 -> 38.1 | Helped |
| Use adaptive temporal aggregation instead of max pooling (ablation) | ROSA temporal aggregation · VISOR-NVOS test | J &F: 37.1 -> 38.1 | Helped |
Surprising findings
- Simple out-of-the-box SAM+CLIP outperforms the CoMMa+SAM adaptation, which is trained on Ego4D narrations.
- Zero-shot ROSA outperforms GroundedSAM, which was trained with bounding box supervision.
Limitations
- Inference is performed per frame, ignoring temporal information.
- Training and evaluation focus on the cooking domain; a performance gap remains on non-cooking videos.
- Performance on VISOR-NVOS lags well behind the SAM upper bound, and plurals are handled poorly.
Experimental setup
Datasets: Ego4D VISOR VOST YouCook2-BB
Benchmarks: VISOR-NVOS VOST YouCook2-BB
Metrics: J (IoU) F (boundary F-measure) J&F J_union J_ins box accuracy point accuracy
Baselines: SAM+CLIP CoMMa+SAM ODISE GroundedSAM CoMMa
Builds on: CLIP SAM CoMMa MIL-NCE
Closest papers by meaning
- Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention (NeurIPS 2025 · similarity 0.89)
- Object-Shot Enhanced Grounding Network for Egocentric Video (CVPR 2025 · similarity 0.87)
- Weakly Supervised Instance Segmentation for Videos With Temporal Mask Consistency (CVPR 2021 · similarity 0.87)
- Point-VOS: Pointing Up Video Object Segmentation (CVPR 2024 · similarity 0.87)
- Mask Grounding for Referring Image Segmentation (CVPR 2024 · similarity 0.86)
- Retrieval-Augmented Egocentric Video Captioning (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.