Research map / Video / Video-language understanding
Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Multi-Pair temporal sentence grounding (MP-TSG), co-training multiple video-query pairs via a Multi-Thread Knowledge Transfer Network (MKTN) with cross-sentence mining, adaptive negative selection, and object-phrase/activity-sentence prototype alignment.
Problem
Temporal sentence grounding methods train each video-query pair separately, ignoring relationships between pairs, which wastes computation and misses shared semantics that could aid grounding.
Approach
Co-trains multiple video-query pairs in one multi-thread framework that transfers knowledge across queries and videos via query-to-query and video-to-query relations, a dynamic cosine-annealed negative-selection threshold, and prototype alignment at object-phrase (spatial) and activity-sentence (temporal) levels.
Key results
- On Charades-STA, MKTN reaches R@1 IoU=0.5 of 69.93 vs 62.50 for the best listed fully-supervised baseline (MRTNet).
- On ActivityNet Captions, R@1 IoU=0.5 is 58.32 vs 48.59 for MMN (fully supervised).
- As a plug-and-play module, it raises 2D-TAN's Charades-STA R@1 IoU=0.5 from 39.81 to 43.37 and VPS from 12.43 to 46.28.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add cross-sentence semantic mining (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 53.82 -> 58.32 | Helped |
| add object-phrase prototype matching (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 55.40 -> 58.32 | Helped |
| add activity-sentence prototype alignment (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 56.27 -> 58.32 | Helped |
| add adaptive video-query matching (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 54.95 -> 58.32 | Helped |
| add temporal relationship modeling to CSM (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 56.40 -> 58.32 | Helped |
| add contextual relationship modeling to CSM (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 57.14 -> 58.32 | Helped |
| add semantic relationship modeling to CSM (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 54.85 -> 58.32 | Helped |
| use both visual and textual features for cross-modal contrast (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 53.03 -> 58.32; 52.84 -> 58.32 | Helped |
| use an adaptive threshold instead of a fixed threshold (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 54.08 -> 58.32 | Helped |
| use the cosine schedule for negative selection (ablation) | MKTN · ActivityNet Captions | R@1, IoU=0.5: 55.15 -> 58.32 | Helped |
| construct appearance prototypes (ablation) | Object-phrase prototype matching in MKTN · ActivityNet Captions | R@1, IoU=0.5: 56.80 -> 58.32 | Helped |
| add object-phrase cross-modal alignment (ablation) | Object-phrase prototype matching in MKTN · ActivityNet Captions | R@1, IoU=0.5: 57.03 -> 58.32 | Helped |
Limitations
- Gains compared against prior single-pair baselines that may not share the same setting; the MP-TSG comparison is not like-for-like.
- Relies on datasets where videos/queries share semantics; benefit on dissimilar pairs is not shown.
Experimental setup
Datasets: ActivityNet Captions Charades-STA TACoS
Benchmarks: ActivityNet Captions Charades-STA TACoS
Metrics: R@1 IoU=0.5 R@1 IoU=0.7 R@5 IoU=0.5 R@5 IoU=0.7 VPS
Baselines: DRN 2D-TAN MIGCN MMN VDI MRTNet MESM LCNet VCA ICVC WSTAN CPL
Builds on: 2D-TAN MMN LCNet Skip-thought GloVe RINCE
Closest papers by meaning
- Temporal Sentence Grounding with Relevance Feedback in Videos (NeurIPS 2024 · similarity 0.91)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.90)
- Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding (AAAI 2024 · similarity 0.89)
- Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding (CVPR 2024 · similarity 0.88)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.88)
- Cross-modal Causal Relation Alignment for Video Question Grounding (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.