Research map / Video / Video-language understanding
SnAG: Scalable and Accurate Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows late cross-modal fusion with video-centric sampling makes single-stage video grounding scalable to long videos with many queries, yielding SnAG, which beats CONE and SOONet on MAD and other benchmarks.
Problem
Temporal video grounding methods are optimized for few queries in short videos and do not scale to long videos with hundreds of text queries; early cross-modal fusion makes inference and training costly.
Approach
Analyzes computational cost of early vs. late fusion, showing late fusion amortizes video encoding across queries; proposes video-centric mini-batch sampling that reuses video features across many queries in training, implemented in a single-stage Transformer (SnAG) with cross-attention late fusion and point-based moment decoding.
Key results
- On MAD, SnAG reaches 13.75% R@5 at tIoU=0.5, 43% more accurate than CONE and 1.5x faster (45.3 queries/s on a P100).
- On Charades-STA with I3D features, SnAG reaches 46.26% R@1 at tIoU=0.7, 3.61 points above prior SOTA SSRN.
- On TACoS, SnAG achieves 44.86% R@1 and 70.66% R@5 at tIoU=0.5, beating MATN by 7.5 and 12.1 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use late fusion with video-centric training (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 44.51 -> 44.86 | Helped |
| Use late fusion with video-centric training (ablation) | SnAG · MAD | Recall@5, tIoU=0.5: 13.51 -> 13.75 | Helped |
| Use video-centric rather than query-centric training (ablation) | SnAG · MAD | training time: 55% | Helped |
| Use video-centric rather than query-centric inference (ablation) | SnAG · MAD | test time: 65% | Helped |
| Remove cross-modal fusion (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 44.94 -> 7.85 | Hurt |
| Move fusion before the video pyramid (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 44.94 -> 44.01 | No clear effect |
| Fuse before and after the video pyramid (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 44.94 -> 44.54 | No clear effect |
| Repeat fusion within the video pyramid (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 44.94 -> 40.79 | Hurt |
| Use token-wise sentence embeddings with cross-attention rather than global embeddings (ablation) | SnAG · TACoS | Recall@1, tIoU=0.5: 39.24 -> 45.06 | Helped |
| Use affine cross-attention rather than vanilla cross-attention (ablation) | SnAG · TACoS | Recall@5, tIoU=0.5: 70.08 -> 70.66 | Helped |
| Increase the number of queries sampled per video to Bq=32 (ablation) | SnAG · MAD | test accuracy | No clear effect |
| Increase the input snippet length (ablation) | SnAG · MAD | accuracy | Helped |
Surprising findings
- Late fusion achieves accuracy comparable to early-fusion variants on TACoS, challenging the conventional wisdom favoring early fusion.
- Video-centric training with batch of a single video (Bq=32) leaves MAD test accuracy unaffected while cutting training time and GPU memory.
Limitations
- Relies on pre-extracted video and text features (C3D, CLIP, BERT, EgoVLP) rather than end-to-end raw-frame training.
- Short-video benchmarks (Charades-STA, ActivityNet-Captions) are known to have temporal biases; results there are competitive rather than uniformly best.
Experimental setup
Datasets: Ego4D-NLQ MAD TACoS Charades-STA ActivityNet-Captions
Benchmarks: Ego4D-NLQ MAD TACoS Charades-STA ActivityNet-Captions
Metrics: R@k at tIoU threshold training time GPU memory inference time
Baselines: CONE SOONet VLG-Net CLIP 2D-TAN VSLNet SSRN MATN G2L
Builds on: ActionFormer FCOS-style point-based detection Soft-NMS Focal loss DIoU loss
Closest papers by meaning
- SimVG: A Simple Framework for Visual Grounding with Decoupled Multi-modal Fusion (NeurIPS 2024 · similarity 0.90)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.89)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.89)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.88)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.88)
- Local-Global Multi-Modal Distillation for Weakly-Supervised Temporal Video Grounding (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.