Research map / Video / Video-language understanding
Temporal Sentence Grounding with Relevance Feedback in Videos
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces the TSG-RF task (temporal sentence grounding that also reports whether a query-related segment exists), reconstructs Charades-STA and ActivityNet Captions into Charades-RF and ActivityNet-RF, and proposes RaTSG, a unified multi-granularity relevance discriminator plus relation-aware grounding network.
Problem
Standard temporal sentence grounding assumes the query-related segment always exists in the video, so models output boundaries even when no relevant content is present, causing erroneous grounding.
Approach
Reformulates grounding as foreground-background frame detection and jointly learns frame-level (max-pooled MIL) and video-level relevance scores; the video-level relation vector is injected as a special index-0 token so the boundary predictor can output "no segment" selectively.
Key results
- On Charades-RF, RaTSG reaches 76.85% relevance accuracy and 59.93 mIoU, versus 71.94% accuracy and 56.23 mIoU for the best cascaded baseline EAMAT++.
- On ActivityNet-RF, RaTSG attains 84.27% accuracy and 61.15 mIoU, outperforming SeqPAN++ (81.60%, 59.11 mIoU).
- RaTSG uses 1.27M parameters, far fewer than cascaded EAMAT++ (98.30M) and UniVTG++ (45.53M).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RaTSG instead of the strongest adapted baseline, EAMAT++ | TSG-RF · Charades-RF | mIoU: 56.23 -> 59.93 | Helped |
| Use RaTSG instead of the strongest adapted baseline, SeqPAN++ | TSG-RF · ActivityNet-RF | mIoU: 59.11 -> 61.15 | Helped |
| Add fine-grained frame-level discrimination to coarse-grained discrimination (ablation) | RaTSG · Charades-RF | mIoU: 59.18 -> 59.93 | Helped |
| Add coarse-grained video-level discrimination to fine-grained discrimination (ablation) | RaTSG · Charades-RF | mIoU: 59.27 -> 59.93 | Helped |
| Use the predicted relation signal vector g instead of a random vector (ablation) | RaTSG segment grounding · Charades-RF | mIoU: 57.82 -> 59.93 | Helped |
| Add the segment grounding module (ablation) | RaTSG relevance discrimination · Charades-RF | Acc: 75.59 -> 76.85 | Helped |
| Add the relevance discriminator (ablation) | RaTSG segment grounding · Charades-STA | mIoU: 49.37 -> 53.02 | Helped |
| Add the multi-granularity relevance discriminator and relation-aware grounding to EAMAT (ablation) | EAMAT on Charades-RF · Charades-RF | mIoU: 56.23 -> 59.58 | Helped |
Surprising findings
- CLIP-based relevance discrimination performs poorly (about 60-63% accuracy), attributed to domain shift.
- On the standard TSG task (without non-grounding samples), RaTSG is not consistently best, trailing EAMAT on R1@0.3 and SeqPAN on R1@0.5.
Limitations
- Fails on audio-related actions (e.g., sneezing) because only visual features are used.
- Misjudges temporal order of actions (e.g., opening vs closing), struggling with temporal-sensitive content.
- Non-grounding samples are constructed synthetically by random video pairing, which may not reflect real-world distributions.
Experimental setup
Datasets: Charades-STA ActivityNet Captions
Benchmarks: Charades-RF ActivityNet-RF Charades-STA ActivityNet Captions
Metrics: Acc R1@0.3 R1@0.5 R1@0.7 mIoU
Baselines: VSLNet SeqPAN EAMAT ADPN UniVTG QD-DETR CLIP-Avg CLIP-Top5
Builds on: VSLNet EAMAT Multiple instance learning max-pooling (Following [20])
Closest papers by meaning
- Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network (AAAI 2025 · similarity 0.91)
- Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding (NeurIPS 2025 · similarity 0.91)
- Number it: Temporal Grounding Videos like Flipping Manga (CVPR 2025 · similarity 0.88)
- VideoGrounding-DINO: Towards Open-Vocabulary Spatio-Temporal Video Grounding (CVPR 2024 · similarity 0.87)
- Learning Multi-Scale Video-Text Correspondence for Weakly Supervised Temporal Article Gronding (AAAI 2024 · similarity 0.87)
- Universal Video Temporal Grounding with Generative Multi-modal Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.