Research map / Video / Video-language understanding
Siamese Learning with Joint Alignment and Regression for Weakly-Supervised Video Paragraph Grounding
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SiamGTR, a siamese transformer that jointly learns cross-modal alignment and temporal boundary regression for weakly-supervised video paragraph grounding using pseudo videos built by splicing query videos into background videos.
Problem
Video paragraph grounding requires costly temporal annotations for multiple sentences; the weakly-supervised setting (WSVPG) for paragraph-level localization has been understudied, and prior weakly-supervised grounding relies on two-stage proposal ranking.
Approach
A weight-sharing siamese transformer with an Augmentation Branch that regresses paragraph boundaries in pseudo videos (query video inserted into a background video, giving pseudo boundary labels) and an Inference Branch that learns order-guided sentence-to-video attention on normal videos, yielding one-stage localization without timestamps.
Key results
- On ActivityNet-Captions under weak supervision, SiamGTR reaches 52.32 mIoU and 57.23 R@0.5, versus 38.54 mIoU for WSTPN.
- SiamGTR weakly-supervised outperforms WSTPN by 11.16 mIoU on TACoS (20.48 vs 9.32) and by 3.72 mIoU on Charades-CD-OOD.
- Semi-supervised SiamGTR outperforms prior semi-supervised methods and is on par with or better than fully-supervised SVPTR on all three datasets (54.57 mIoU on ActivityNet-Captions).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add random boundary shifting and random re-sampling for pseudo-data generation (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 44.46 -> 52.32 | Helped |
| Add random boundary shifting (ablation) | SiamGTR pseudo-data generation · ActivityNet-Captions | mIoU: 46.04 -> 46.96 | Helped |
| Add random re-sampling (ablation) | SiamGTR pseudo-data generation · ActivityNet-Captions | mIoU: 46.04 -> 50.94 | Helped |
| Add modulated positional encodings (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 44.46 -> 48.52 | Helped |
| Add the conceptual semantic connector (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 50.06 -> 52.32 | Helped |
| Add dynamic anchor boxes (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 48.75 -> 52.32 | Helped |
| Use self-consistent regression loss instead of plain regression loss (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 50.62 -> 52.32 | Helped |
| Add order-guided attention loss (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 31.82 -> 52.32 | Helped |
| Add cross-branch loss (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 48.07 -> 52.32 | Helped |
| Add anchor ranking loss (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 49.14 -> 52.32 | Helped |
| Add pseudo attention loss (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 51.76 -> 52.32 | Helped |
| Use a learnable paragraph representation instead of max-pooling (ablation) | SiamGTR · ActivityNet-Captions | mIoU: 49.55 -> 52.32 | Helped |
Surprising findings
- Weakly-supervised SiamGTR is comparable to or better than some semi-supervised baselines that use up to 10% temporal labels.
Limitations
- Performance on TACoS remains well below fully-supervised methods (20.48 vs 40+ mIoU range for FS models).
- Pseudo videos from splicing may introduce synthesis artifacts and boundary uncertainty, mitigated only heuristically by random boundary shifting.
Experimental setup
Datasets: ActivityNet-Captions Charades-CD-OOD TACoS
Benchmarks: ActivityNet-Captions Charades-CD-OOD TACoS
Metrics: R@0.3 R@0.5 R@0.7 mIoU R@0.1
Baselines: WSSL WSTPN DepNet PRVG SVPTR HSCNet VPTR 3D-TPN STLG
Builds on: DETR Dynamic-DETR (anchor boxes) MoCo WSTAN Video Paragraph Grounding (Bao et al.)
Closest papers by meaning
- Semi-Supervised Video Paragraph Grounding With Contrastive Encoder (CVPR 2022 · similarity 0.90)
- Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding (AAAI 2024 · similarity 0.90)
- ProTeGe: Untrimmed Pretraining for Video Temporal Grounding by Video Temporal Grounding (CVPR 2023 · similarity 0.90)
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.88)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.88)
- Omnipotent Distillation with LLMs for Weakly-Supervised Natural Language Video Localization: When Divergence Meets Consistency (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.