Research map / Video / Video-language understanding
Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BTDP, which uses visual/semantic boundary segmentation and an LLM with self-evaluation context aggregation to generate human-like pseudo moment-query pairs for training an EMB localizer in zero-shot video grounding.
Problem
Zero-shot natural language video localization needs pseudo moment-query pairs, but existing methods produce simplistic word-level queries that miss real-world query complexity and suffer from insensitivity, isolation, and lack of regulation when LLMs are naively integrated.
Approach
Combine visual boundary detection with LLM-based semantic boundary segmentation over keyframe captions/tags, then aggregate atomic segments with a global-context LLM prompt that iteratively self-evaluates against explicit criteria before producing pseudo-supervision pairs.
Key results
- On Charades-STA, BTDP reaches 39.05 mIoU and 58.31 R@0.3, the best among zero-shot methods and a 4.18% mIoU gain over the second-best method.
- On ActivityNet Captions, BTDP achieves 36.55 mIoU and 50.64 R@0.3, outperforming other zero-shot methods on R@0.3, R@0.7, and mIoU.
- Ablation on Charades-STA shows boundary segmentation and context aggregation together raise mIoU from 33.95 to 38.19; self-evaluation adds about 0.86 mIoU (38.19 to 39.05).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add semantic boundary segmentation (ablation) | BTDP · Charades-STA | mIoU: 33.95 -> 34.81; 36.71 -> 38.19 | Helped |
| add context aggregation (ablation) | BTDP · Charades-STA | mIoU: 33.95 -> 36.71; 34.81 -> 38.19 | Helped |
| add self-evaluation (ablation) | BTDP · Charades-STA | R@0.5: 38.47 -> 40.00 | Helped |
| include keyframe tags (ablation) | BTDP · Charades-STA | mIoU: 37.24 -> 38.19 | Helped |
| use pseudo-supervision pairs to supplement missing manual annotations (ablation) | BTDP · Charades-STA | performance | Helped |
| use BTDP instead of LoVe | EMB trained for zero-shot localization · ActivityNet Captions | R@0.5: 31.45 -> 30.56 | Hurt |
Surprising findings
- Using only 70% of manual annotations plus pseudo data is reported to retain high accuracy, while removing pseudo data degrades performance sharply.
Limitations
- Relies on proprietary Gemini LLM, with no open-model comparison.
- Pseudo queries may not precisely match fine-grained query-segment boundaries, as noted by the authors' future work.
- Self-evaluation threshold and iteration details are described only qualitatively.
Experimental setup
Datasets: Charades-STA ActivityNet Captions
Benchmarks: Charades-STA ActivityNet Captions
Metrics: R@0.3 R@0.5 R@0.7 mIoU
Baselines: PSVL CORONET LoVe ChatVTG VTimeLLM 2D-TAN EMB SCN CRM CNM
Builds on: EMB PSVL LoVe RAM Katna
Closest papers by meaning
- Commonsense for Zero-Shot Natural Language Video Localization (AAAI 2024 · similarity 0.89)
- Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · similarity 0.88)
- Zero-shot Video Moment Retrieval via Off-the-shelf Multimodal Large Language Models (AAAI 2025 · similarity 0.88)
- Transferable Video Moment Localization by Moment-Guided Query Prompting (AAAI 2024 · similarity 0.88)
- Masked Diffusion Vision-Language Models for Temporal Action Localization (NeurIPS 2026 · similarity 0.87)
- Unsupervised Domain Adaptative Temporal Sentence Localization with Mutual Information Maximization (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.