Research map / Video / Video-language understanding
Exploiting Auxiliary Caption for Video Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ACNet, which mines auxiliary captions from dense video captions (via NACS) and uses caption-guided attention and asymmetric cross-modal contrastive learning to improve video moment grounding under sparse annotations.
Problem
Video grounding datasets have sparse moment-sentence annotations, so unannotated actions that relate to queries are ignored, and naively adding dense captions as data augmentation degrades performance.
Approach
Filter dense captions from PDVC into non-overlapping high-quality auxiliary captions (NACS), use them as temporal priors via caption-guided attention, and apply asymmetric contrastive learning that pulls only ground-truth pairs and treats auxiliary-caption sentences as hard negatives for ground-truth moments.
Key results
- On ActivityNet Captions, ACNet reaches R@1 IoU=0.5 of 56.39 with manual annotations from other moments at inference (ACNet^natural) and 52.51 for the base ACNet, versus 52.89 for SPL (the strongest prior result listed).
- On TACoS, ACNet achieves R@1 IoU=0.5 of 36.79 (ACNet) and 38.84 (natural variant), compared with 37.57 for MATN.
- On ActivityNet-CG test-trivial, ACNet reaches R@1 IoU=0.7 of 33.52 versus 29.64 for VISA.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use ACNet instead of the strongest listed baseline | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 52.89 -> 56.39 | Helped |
| Use ACNet instead of the strongest listed baseline | TACoS · TACoS | R@1 at IoU=0.3: 48.79 -> 51.64 | Helped |
| Use ACNet instead of VISA | ActivityNet-CG · ActivityNet-CG Test-Trivial | R@1 at IoU=0.5: 47.13 -> 51.81 | Helped |
| Remove NACS (ablation) | ACNet · ActivityNet-CG Test-Trivial and Novel-Comp | R@1 at IoU=0.5 and 0.7: 51.81 -> 46.33; 33.30 -> 30.71 | Hurt |
| Add NACS to the ACCL-only model (ablation) | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 47.28 -> 51.83 | Helped |
| Add CGA to the regression-only model (ablation) | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 46.74 -> 52.08 | Helped |
| Add NACS to the two-branch ACCL+Reg model (ablation) | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 50.24 -> 54.85 | Helped |
| Add CGA to the two-branch ACCL+Reg model (ablation) | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 50.24 -> 55.27 | Helped |
| Add NACS and CGA to the ACCL+Reg model (ablation) | ActivityNet Captions · ActivityNet Captions | R@1 at IoU=0.5: 50.24 -> 56.39 | Helped |
| Remove APN from ACCL (ablation) | ACNet · ActivityNet Captions | R@1 at IoU=0.5: 56.39 -> 53.79 | Hurt |
| Remove ANDM from ACCL (ablation) | ACNet · ActivityNet Captions | R@1 at IoU=0.5: 56.39 -> 53.58 | Hurt |
Surprising findings
- Naively adding dense captions as data augmentation degrades 2D-TAN performance (drop of 1.01% with 10 extra captions).
- ACNet without NACS is weaker than VISA on all ActivityNet-CG splits, indicating auxiliary captions are crucial for generalization.
Limitations
- Higher training and inference cost than prior 2D-TAN and MMN methods.
- Auxiliary captions are coarse and may overlap ground truth, requiring hand-designed asymmetric losses.
- The strongest TACoS and ActivityNet results with the natural variant use manual annotations from other moments at inference, so it is not a fully fair comparison.
Experimental setup
Datasets: ActivityNet Captions TACoS ActivityNet-CG
Benchmarks: ActivityNet Captions TACoS ActivityNet-CG
Metrics: R@1 IoU=0.3 R@1 IoU=0.5 R@1 IoU=0.7 R@5 IoU=0.3 R@5 IoU=0.5 R@5 IoU=0.7
Baselines: CTRL CBP SCDM 2D-TAN DRN FVMR RaNet DPIN MATN CBLN SMIN GTR
Builds on: 2D-TAN PDVC MMN DRN
Closest papers by meaning
- Comprehensive Visual Grounding for Video Description (AAAI 2024 · similarity 0.90)
- Explicit Temporal-Semantic Modeling for Dense Video Captioning via Context-Aware Cross-Modal Interaction (AAAI 2026 · similarity 0.88)
- Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval? (CVPR 2023 · similarity 0.88)
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.87)
- AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark (ICLR 2025 · similarity 0.86)
- Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval (CVPR 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.