Research map / Video / Video-language understanding
VVS: Video-to-Video Retrieval with Irrelevant Frame Suppression
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes VVS, an end-to-end video-level retrieval network that suppresses irrelevant frames via an easy distractor elimination stage and a suppression weight generation stage (temporal saliency + topic guidance), reaching frame-level-like accuracy with video-level speed.
Problem
Video-level features for content-based video retrieval are fast but inaccurate because a single feature struggles to represent long untrimmed videos containing distractor frames.
Approach
Explicitly suppress irrelevant frames before pooling: first remove easy distractors (low-magnitude L_N-iMAC features) with a learned discrimination module, then weight remaining frames by temporal saliency (from frame-similarity maps) and topic relevance (cosine similarity to a rough topic vector), aggregating into one video-level feature.
Key results
- VVS_3840 reaches 0.711 DSVR, 0.689 CSVR and 0.590 ISVR mAP on FIVR-200K, about 25% above the best video-level baseline (TCA_c at 0.570/0.553/0.473 for the first two tasks shown)
- On CC_WEB_VIDEO, VVS_3840 achieves 0.975 cc and 0.984 cc_c, the best video-level results, close to frame-level ViSiL_v
- Ablation on FIVR-5K: combining DDM, TSM and TGM raises ISVR from 0.651 baseline to 0.705
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add VVS frame suppression (DDM, TSM, and TGM) | Video-level retrieval features · FIVR-200K ISVR | mAP: 0.473 -> 0.590 | Helped |
| Add DDM easy-distractor elimination (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.651 -> 0.672 | Helped |
| Add TSM temporal saliency weighting (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.651 -> 0.661 | Helped |
| Add TGM topic guidance (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.651 -> 0.677 | Helped |
| Use all three modules DDM, TSM, and TGM (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.651 -> 0.705 | Helped |
| Set DDM injection ratio to 20% - 50% rather than 0% - 20% (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.701 -> 0.705 | Helped |
| Increase DDM injection ratio to 80% - 100% rather than 20% - 50% (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.705 -> 0.701 | Hurt |
| Add frame loss to TSM (ablation) | TSM · FIVR-5K | ISVR mAP: 0.702 -> 0.705 | Helped |
| Use topic representation G instead of a random or constant initial state (ablation) | TGM · FIVR-5K | ISVR mAP: 0.652 -> 0.677 | Helped |
| Add refinement and hierarchical connection to topic initialization (ablation) | TGM · FIVR-5K | ISVR mAP: 0.584 -> 0.677 | Helped |
| Suppress easy distractors with trained DDM instead of manual inference-time magnitude thresholding (ablation) | VVS3840 · FIVR-5K | ISVR mAP: 0.665 -> 0.672 | Helped |
Surprising findings
- Oracle experiments with manual removal of annotated distractors boost prior video-level methods' ISVR by 1.24x to 1.58x, suggesting suppression is the key bottleneck
- Training the distractor module outperforms simple manual magnitude thresholding at inference, which degrades with threshold changes
Limitations
- Some EVVE videos could not be downloaded, so the EVVE benchmark is only reported on a subset in the supplement
- Backbone L_N-iMAC is kept frozen, so feature learning is limited to the suppression modules and PCA-reduced features
- Several hyperparameters (magnitude threshold 40, injection ratio, tempered sigmoid temperature) are tuned on FIVR-5K used as validation
Experimental setup
Datasets: VCDB FIVR-5K FIVR-200K CC_WEB_VIDEO
Benchmarks: FIVR-200K FIVR-5K CC_WEB_VIDEO
Metrics: mAP queries processed per 10 minutes
Baselines: TN DP TCA TNIP ViSiL CTE HC DML TMK LAMV VRAG
Builds on: ViSiL TCA R-MAC Chamfer Similarity VCDB
Closest papers by meaning
- MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondence (NeurIPS 2025 · similarity 0.86)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.85)
- Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension (NeurIPS 2025 · similarity 0.85)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.85)
- CoVR: Learning Composed Video Retrieval from Web Video Captions (AAAI 2024 · similarity 0.85)
- BOLT: Boost Large Vision-Language Model Without Training for Long-form Video Understanding (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.