Research map / Video / Video-language understanding
Commonsense for Zero-Shot Natural Language Video Localization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CORONET, a zero-shot natural language video localization model that injects ConceptNet commonsense via a GCN-encoded concept graph and cross-attention enhancement of video and pseudo-query features.
Problem
Zero-shot natural language video localization relies on pseudo-queries built from detected objects and verbs that are poorly grounded in the video, producing noisy, disjointed supervision.
Approach
A Commonsense Enhancement Module builds a seed-concept subgraph from ConceptNet (filtered relation types), encodes it with a GCN conditioned on the video, and uses separate cross-attention enhancement of video and pseudo-query features before cross-modal localization.
Key results
- On Charades-STA, CORONET reaches R@0.5 34.60 and mIoU 32.73, versus PSVL's 30.84 and 31.09 (reproduced).
- Gains of up to 32.13% over PSVL in higher recall thresholds on Charades-STA, and 7.02% to 17.15% across metrics on ActivityNet-Captions.
- On ActivityNet-Captions, CORONET reaches R@0.5 28.19 and R@0.3 46.05 versus LFVL's 43.34 and 25.17 at R@0.5 (reproduced).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use CORONET instead of LFVL | zero-shot NLVL · ActivityNet-Captions | R@0.5: 25.17 -> 28.19 | Helped |
| use CORONET instead of LFVL | zero-shot NLVL · Charades-STA | R@0.7: 16.95 -> 17.93 | Helped |
| use 250 rather than 300 seed concepts (ablation) | CORONET · Charades-STA | R@0.3: 49.21 -> 50.98 | Helped |
| use 250 rather than 300 seed concepts (ablation) | CORONET · Charades-STA | R@0.7: 17.93 -> 16.48 | Hurt |
| use spatial relations only (ablation) | CORONET · Charades-STA | mIoU: 32.73 -> 26.03 | Hurt |
| add spatial relations to temporal relations (ablation) | CORONET · Charades-STA | R@0.3: 44.98 -> 49.84 | Helped |
| remove spatial and temporal relation types from filtered relations (ablation) | CORONET · Charades-STA | mIoU: 32.73 -> 30.77 | Hurt |
| add all ConceptNet relation types instead of filtered relations (ablation) | CORONET · Charades-STA | mIoU: 32.73 -> 32.85 | Helped |
| enhance video and query features separately rather than sharing enhancement weights (ablation) | CORONET · Charades-STA | R@0.3: 40.21 -> 49.21 | Helped |
| enhance query only, omitting video enhancement (ablation) | CORONET · Charades-STA | R@0.5: 34.60 -> 22.16 | Hurt |
| enhance video only, omitting query enhancement (ablation) | CORONET · Charades-STA | R@0.3: 49.21 -> 44.11 | Hurt |
| replace GRU input encoders with Transformer encoders (ablation) | CORONET · Charades-STA | R@0.3: 49.21 -> 53.57 | Helped |
Surprising findings
- Using a multi-relational RGCN graph instead of a weighted directed graph hurt performance, and expanding the concept graph to 1-hop neighbors worsened results.
- Adding CEM to the language-free LFVL baseline gave mixed results: post-fusion improved mIoU and R@0.3 but reduced R@0.5 and R@0.7.
Limitations
- Gains over PSVL and LFVL are mixed across metrics; some baselines win on R@0.7 or mIoU for certain configurations.
- Trade-off between recall at high thresholds and mIoU depending on concept set size.
- Relies on an off-the-shelf object detector and noun-only pseudo-queries, so the commonsense benefit is bounded by detector quality.
Experimental setup
Datasets: Charades-STA ActivityNet-Captions VisualGenome ConceptNet
Benchmarks: Charades-STA ActivityNet-Captions
Metrics: R@0.3 R@0.5 R@0.7 mIoU
Baselines: CTRL LGI TGA WSTG SCN WS-DEC WSLLN PSVL LFVL
Builds on: PSVL LFVL Attention Dynamic Filters ConceptNet
Closest papers by meaning
- Boundary-Aware Temporal Dynamic Pseudo-Supervision Pairs Generation for Zero-Shot Natural Language Video Localization (AAAI 2025 · similarity 0.89)
- Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · similarity 0.88)
- Localizing Events in Videos with Multimodal Queries (CVPR 2025 · similarity 0.86)
- Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding (NeurIPS 2025 · similarity 0.85)
- Dense Video Object Captioning from Disjoint Supervision (ICLR 2025 · similarity 0.85)
- VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning (AAAI 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.