Research map / Video / Video-language understanding
Anchor-Aware Similarity Cohesion in Target Frames Enables Predicting Temporal Moment Boundaries in 2D
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes A2FA, which uses a CLIP-based anchor frame and DTW-derived anchor-aware mask to align query and frame features, and F2SGD, which predicts moment boundaries as a single top-right vertex detection in the 2D frame-frame similarity matrix.
Problem
Video moment retrieval must align query text with the relevant frames, avoiding irrelevant frames, and then map aligned frame features to start and end boundaries despite the information gap between frame semantics and temporal coordinates.
Approach
Identify an anchor frame via CLIP query-frame similarity, build an anchor-aware mask from DTW-based semantic transition points to restrict query alignment to a compact segment, and then predict boundaries as a single point detection (top-right corner of high-similarity blocks) in the 2D frame-frame similarity matrix.
Key results
- On QVHighlights test, the method reaches R1@0.7 of 64.3 and mAP@Avg of 63.4, surpassing prior CLIP-based SOTA by 12.2% on R1@0.7 (as reported by the authors).
- On ActivityNet Captions, R1@0.5 of 71.1 and R1@0.7 of 43.0, versus Mr. BLIP's 53.9 and 35.6.
- On Charades-STA, R1@0.5 of 83.8 versus 69.3 for Mr. BLIP (BLIP-2 backbone); the authors note the gain is smaller than on other datasets.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the full A2FA and F2SGD method instead of the strongest reported baseline | Video moment retrieval · QVHighlights test | R1@.7: 60.5 -> 64.3 | Helped |
| Use the full method instead of Mr. BLIP | Video moment retrieval · ActivityNet Captions | R1@.7: 35.6 -> 43.0 | Helped |
| Use the full method instead of Mr. BLIP | Video moment retrieval · Charades-STA test | R1@.7: 49.2 -> 50.1 | Helped |
| Add F2SGD to query-frame similarity-only retrieval (ablation) | Video moment retrieval on QVHighlights · QVHighlights val | R1@.7: 5.2 -> 39.6 | Helped |
| Add A2FA alignment to F2SGD (ablation) | Video moment retrieval on QVHighlights · QVHighlights val | R1@.7: 39.6 -> 57.6 | Helped |
| Add A2FA reweighting (ablation) | Video moment retrieval on QVHighlights · QVHighlights val | R1@.7: 57.6 -> 64.8 | Helped |
| Use BLIP-2 rather than CLIP to generate the anchor-aware mask (ablation) | A2FA · QVHighlights val | R1@.7: 64.8 -> 66.0 | Helped |
| Use BLIP-2 rather than CLIP to generate the anchor-aware mask (ablation) | A2FA · QVHighlights val | mAP@Avg: 63.0 -> 65.1 | Helped |
| Use the coarse [bL, bR] interval without final F2SGD refinement (ablation) | Video moment retrieval · QVHighlights val | R1@.7: 23.8 -> 64.8 | Hurt |
| Use the coarse [bL, bR] interval without final F2SGD refinement (ablation) | Video moment retrieval · QVHighlights val | mAP@Avg: 31.4 -> 63.0 | Hurt |
| Set F2SGD kernel size to 17 rather than 21 (ablation) | F2SGD · QVHighlights val | R1@.7: 63.9 -> 64.8 | Hurt |
| Set F2SGD kernel size to 25 rather than 21 (ablation) | F2SGD · QVHighlights val | R1@.7: 64.1 -> 64.8 | Hurt |
Surprising findings
- Replacing the alignment module of Moment-DETR and UniVTG with A2FA raised R1@0.7 substantially (e.g., Moment-DETR 34.1 to 49.1), indicating the alignment is pluggable.
Limitations
- Gains on Charades-STA are less significant, attributed to shorter videos making localization harder.
- The anchor-aware mask depends on the CLIP query-frame similarity to find the anchor frame; the interval from Eq. (5) alone is a coarse estimate (R1@0.7 23.8) and needs the F2SGD refinement.
Experimental setup
Datasets: QVHighlights ActivityNet Captions Charades-STA
Benchmarks: QVHighlights ActivityNet Captions Charades-STA
Metrics: R1@0.5 R1@0.7 mAP@0.75 mAP@Avg
Baselines: Moment DETR QD-DETR EaTR TR-DETR UVCOM CG-DETR QD-VMR BAM-DETR R2-Tuning Mr. BLIP UnLoc-L UniMD+Sync.
Builds on: CLIP Moment DETR CG-DETR UniVTG
Closest papers by meaning
- MomentSeeker: A Task-Oriented Benchmark For Long-Video Moment Retrieval (NeurIPS 2025 · similarity 0.88)
- Towards Balanced Alignment: Modal-Enhanced Semantic Modeling for Video Moment Retrieval (AAAI 2024 · similarity 0.88)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.87)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.87)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.87)
- Logic-in-Frames: Dynamic Keyframe Search via Visual Semantic-Logical Verification for Long Video Understanding (NeurIPS 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.