Research map / Video / Video-language understanding
Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a frame-level framework for partially relevant video retrieval that decomposes sentences into frame-aligned semantic units (SDM) and adds an adaptive local window aggregator (ALA), plugged into existing PRVR baselines.
Problem
Partially relevant video retrieval (PRVR) on untrimmed videos suffers because frame-level video features are aligned with a single sentence-level representation, a granularity mismatch that prevents frame features from capturing local fine details.
Approach
Decompose the sentence into multiple learnable semantic units via slot attention, each matched to frame-level features and averaged for the frame score; add a multi-scale window-attention Adaptive Local Aggregator for frame features, plus semantic diversity and frame-semantic alignment losses.
Key results
- On TVR, GMMFormer v2 with the proposed framework reaches SumR 196.4 versus 186.9 for the reproduced baseline (R@1 17.4 vs 15.3).
- Applied to MS-SL on TVR, SumR improves from 171.3 to 186.2.
- On ActivityNet Captions, GMMFormer v2 with the framework reaches SumR 156.6 versus 154.6 reproduced baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add the proposed frame-level framework to GMMFormer v2 | GMMFormer v2 · TVR | R@1: 16.2 -> 17.4 | Helped |
| Add the proposed frame-level framework to GMMFormer v2 | GMMFormer v2 · ActivityNet Captions | R@1: 8.9 -> 9.1 | Helped |
| Use separate query branches for frame and clip branches (ablation) | GMMFormer v2 · TVR | R@1: 15.3 -> 16.0 | Helped |
| Add Semantic-Decomposed Matching (ablation) | GMMFormer v2 · TVR | R@1: 15.3 -> 17.1 | Helped |
| Add Semantic Diversity loss (ablation) | GMMFormer v2 + SDM · TVR | R@1: 17.1 -> 16.8 | Hurt |
| Add Adaptive Local Aggregator (ablation) | GMMFormer v2 + SDM + LSD · TVR | R@1: 16.8 -> 17.2 | Helped |
| Add Frame-Semantic Alignment loss (ablation) | GMMFormer v2 + SDM + LSD + ALA · TVR | R@1: 17.2 -> 17.4 | Helped |
| Add ALA without SDM (ablation) | GMMFormer v2 with separate query branches · TVR | R@1: 16.0 -> 15.8 | Hurt |
| Increase semantic units from 1 to 5 (ablation) | GMMFormer v2 + Ours · TVR | R@1: 16.2 -> 17.4 | Helped |
| Increase semantic units from 5 to 9 (ablation) | GMMFormer v2 + Ours · TVR | R@1: 17.4 -> 17.8 | No clear effect |
| Replace average aggregation with max pooling over semantic-unit scores (ablation) | GMMFormer v2 + Ours · TVR | R@1: 17.4 -> 16.1 | Hurt |
| Replace average aggregation with softmax weighting over semantic-unit scores (ablation) | GMMFormer v2 + Ours · TVR | R@1: 17.4 -> 16.7 | Hurt |
Surprising findings
- ALA without the semantic decomposition (SDM) degrades performance (SumR 188.2 vs 189.5 for query branch alone), suggesting a single query does not align with locally detailed video features.
Limitations
- Gains on ActivityNet Captions are small (e.g., MS-SL SumR drops slightly from 139.7 to 143.4 is an improvement but GMMFormer v2 only +2.0)
- Requires replacing the frame branch of existing PRVR models, so it is not a standalone architecture
- Number of semantic units and window scales are hyperparameters chosen empirically
Experimental setup
Datasets: TVR ActivityNet Captions
Benchmarks: TVR ActivityNet Captions
Metrics: R@1 R@5 R@10 R@100 SumR
Baselines: MS-SL GMMFormer GMMFormer v2 PEAN DL-DKD XML ReLoCLNet CONQUER
Builds on: MS-SL GMMFormer v2 slot attention InfoNCE triplet ranking loss
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.92)
- Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.90)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.88)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- TOPA: Extending Large Language Models for Video Understanding via Text-Only Pre-Alignment (NeurIPS 2024 · similarity 0.88)
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.