Research map / Video / Video-language understanding
Object-Shot Enhanced Grounding Network for Egocentric Video
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OSGNet, an egocentric video grounding model that adds query-relevant object features and a head-movement-based shot branch with shot-level contrastive learning, achieving SOTA on Ego4D-NLQ, Ego4D-Goal-Step and TACoS.
Problem
Natural language query (NLQ) grounding in untrimmed egocentric videos, where queries often target fine-grained background objects and the wearer's attention shifts are not captured by existing exocentric-oriented or video-text pretrained methods.
Approach
Injects query-relevant fine-grained object features (detected by Co-DETR, encoded with CLIP text encoder) through parallel cross-attention with a gated fusion, and segments videos into shots at head-turning captions (from LAVILA) to train a shot-query contrastive branch.
Key results
- On Ego4D-NLQ v2 test, OSGNet† reaches 27.60 R@1 IoU=0.3 and 19.78 R@1 IoU=0.5 on validation (with NaQ pretraining + E+I features), outperforming GroundVQA by over 1.5 points.
- On TACoS (C3D features), OSGNet achieves 57.57 R@1 IoU=0.3 and 48.18 R@1 IoU=0.5 on validation, a 3.32-point R@1 IoU=0.5 gain over SnAG on the test set.
- On Ego4D-Goal-Step test with order prior, OSGNet* improves R@1 IoU=0.3 by about 3.65 points over BayesianVSLNet.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add object features (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 19.99 -> 22.03 | Helped |
| add shot-level contrastive loss (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 21.64 -> 22.03 | Helped |
| add both shot contrastive loss and object features (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 19.60 -> 22.03 | Helped |
| use R, HM, and RM verbs instead of R verbs for shot segmentation (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 22.03 -> 21.99 | Hurt |
| use R and HM verbs instead of R verbs for shot segmentation (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 22.03 -> 21.84 | Hurt |
| use HM verbs without R for shot segmentation (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.5: 22.03 -> 21.20 | Hurt |
| replace R-only shot segmentation with R and HM verbs (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.3: 31.63 -> 31.22 | Hurt |
| replace image-based object features with text-based object features (ablation) | OSGNet · Ego4D-NLQ v2 validation | R@1 IoU=0.3: 30.98 -> 31.63 | Helped |
| add OSGNet object and shot-enhancement approach | Ego4D-NLQ v2 · Ego4D-NLQ v2 test | R@1 IoU=0.5: 17.63 -> 19.78 | Helped |
| add OSGNet object and shot-enhancement approach | Ego4D-Goal-Step · Ego4D-Goal-Step test | R@1 IoU=0.3: 35.18 -> 38.83 | Helped |
| add OSGNet object and shot-enhancement approach | TACoS · TACoS test | R@1 IoU=0.5: 44.86 -> 48.18 | Helped |
Surprising findings
- Removing the object feature branch reduces R@1 by over 2 points, while text-based object features outperform image-based object features by 0.65 points at R@1 IoU=0.3.
- Head-turning verbs alone (R) work best for shot segmentation, while adding frequent camera-movement verbs did not improve results.
Limitations
- Performance on egocentric datasets remains lower than on exocentric TACoS, attributed to egocentric complexity.
- Object features depend on a pretrained open-vocabulary detector and query nouns, with a confidence threshold hyperparameter.
- Shot segmentation relies on captions from LAVILA describing head movements.
Experimental setup
Datasets: Ego4D-NLQ NaQ pretraining data Ego4D-Goal-Step TACoS LVIS
Benchmarks: Ego4D-NLQ v1 Ego4D-NLQ v2 Ego4D-Goal-Step TACoS
Metrics: R@1 IoU=0.3 R@1 IoU=0.5 R@5 IoU=0.3 R@5 IoU=0.5
Baselines: InternVideo CONE SnAG NaQ RGNet ASL GroundNLQ EgoEnv EgoVideo GroundVQA BayesianVSLNet VSLNet
Builds on: ActionFormer-style multi-scale transformer (GroundNLQ) NaQ pretraining Co-DETR LAVILA BiMamba
Closest papers by meaning
- Grounded-Exo2Ego: Structured Semantic Grounding for Robust Exocentric-to-Egocentric Video Generation (NeurIPS 2026 · similarity 0.88)
- Test-Time Adaptation for Combating Missing Modalities in Egocentric Videos (ICLR 2025 · similarity 0.88)
- Learning to Segment Referred Objects from Narrated Egocentric Videos (CVPR 2024 · similarity 0.87)
- Action Scene Graphs for Long-Form Understanding of Egocentric Videos (CVPR 2024 · similarity 0.87)
- EgoThinker: Unveiling Egocentric Reasoning with Spatio-Temporal CoT (NeurIPS 2025 · similarity 0.87)
- Robust Egocentric Referring Video Object Segmentation via Dual-Modal Causal Intervention (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.