Research map / Video / Video-language understanding
DeCafNet: Delegate and Conquer for Efficient Temporal Grounding in Long Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DeCafNet, which runs a cheap sidekick encoder over all clips to pick top-c% salient clips for an expert encoder, then fuses both feature sets with DeCaf-Grounder for efficient long-video temporal grounding.
Problem
Long video temporal grounding (LVTG) requires running a costly expert video encoder on every clip, which does not scale with video length.
Approach
A "delegate-and-conquer" dual-encoder design: a lightweight sidekick encoder (pooling plus temporal interpolation, trained with saliency contrastive and distillation losses) extracts dense features and a query-saliency map, so the expert encoder only processes the top-c% salient clips; DeCaf-Grounder fuses the two temporal resolutions via query-aware aggregation and multi-scale refinement.
Key results
- On Ego4D-NLQ, DeCafNet-50% reaches 24.44 AVG recall vs. 23.14 for SnAG and 21.81 for RGNet, while cutting encoder TFLOPs by 47%.
- DeCafNet-30% reduces TFLOPs by 66%, GPU memory by 65%, and inference time by 67% relative to processing all clips with the expert encoder.
- On MAD, DeCafNet reaches 16.47 AVG recall vs. 13.84 for SnAG using the same expert features.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use DeCafNet-50% instead of SnAG | Ego4D-NLQ · Ego4D-NLQ validation | AVG recall: 23.14 -> 24.44 | Helped |
| Use DeCafNet-50% instead of RGNet | Ego4D-Goalstep · Ego4D-Goalstep validation | AVG recall: 30.49 -> 31.61 | Helped |
| Use DeCaf-Grounder instead of SnAG | MAD · MAD | AVG recall: 13.84 -> 16.47 | Helped |
| Add expert salient features to dense sidekick features (ablation) | DeCafNet · Ego4D-NLQ validation | AVG recall: 21.51 -> 23.91 | Helped |
| Add saliency scores to combined dense and salient features (ablation) | DeCafNet · Ego4D-NLQ validation | AVG recall: 23.91 -> 24.41 | Helped |
| Replace random clip selection with saliency selection at 30% (ablation) | DeCafNet · Ego4D-NLQ validation | AVG recall: 21.90 -> 23.94 | Helped |
| Replace uniform clip selection with saliency selection at 50% (ablation) | DeCafNet · Ego4D-NLQ validation | AVG recall: 23.34 -> 24.41 | Helped |
| Move convolution pooling from before block 1 to before block 3 (ablation) | Sidekick encoder · Sidekick encoder | TFLOPs: 21.6 -> 53.9 | Hurt |
| Change temporal interpolation stride from τ=2 to τ=3 (ablation) | Sidekick encoder · Sidekick encoder | Recall: 80.5 -> 72.8 | Hurt |
| Remove saliency loss (ablation) | Sidekick encoder · Sidekick encoder | Recall: 80.5 -> 51.1 | Hurt |
| Remove distillation loss (ablation) | Sidekick encoder · Sidekick encoder | Recall: 80.5 -> 48.5 | Hurt |
| Remove query-aware temporal aggregation (ablation) | DeCaf-Grounder · Ego4D-NLQ validation | AVG recall: 24.41 -> 22.37 | Hurt |
Surprising findings
- Using only the sidekick dense features (no expert features) yields much lower recall (AVG 21.51 vs 24.41), indicating pooling information loss matters.
- Robustness to lighting, blur, and occlusion augmentations drops performance only slightly, staying above prior best SnAG.
Limitations
- MAD experiments cannot use the sidekick encoder because only pre-extracted features are provided, so the efficiency component is not validated there.
- Sidekick encoder pooling loses information, so the expert encoder is still needed on a subset of clips.
Experimental setup
Datasets: Ego4D-NLQ Ego4D-Goalstep MAD Charades-STA TACoS
Benchmarks: Ego4D-NLQ Ego4D-Goalstep MAD Charades-STA TACoS
Metrics: Recall@Top-K at IoU threshold (R1@0.3, R1@0.5, R5@0.3, R5@0. TFLOPs GPU memory inference time
Baselines: 2D-TAN UniVTG VSLNet M-DETR SOONet H-Hands CONE RGNet SnAG M-Guide SMIN MATN
Builds on: SnAG RGNet VideoMAE-style spatio-temporal transformer (MViT-style pooling) Focal loss Distance-IoU loss
Closest papers by meaning
- SnAG: Scalable and Accurate Video Grounding (CVPR 2024 · similarity 0.87)
- VTG-LLM: Integrating Timestamp Knowledge into Video LLMs for Enhanced Video Temporal Grounding (AAAI 2025 · similarity 0.86)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.86)
- Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding (NeurIPS 2025 · similarity 0.86)
- Seq2Time: Sequential Knowledge Transfer for Video LLM Temporal Grounding (CVPR 2025 · similarity 0.86)
- ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long Videos (CVPR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.