Research map / Video / Video-language understanding
Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes BiHSSP, a bidirectional hierarchical sliding semantic probe that computes dynamic video-text similarity via cross-correlation with O(n) retrieval cost, as a plug-and-play module on CLIP-based text-video retrieval.
Problem
Text-video retrieval suffers because videos and texts convey unequal information, so fixed cosine similarity is suboptimal, while cross-modal interaction methods that give dynamic similarity cost O(n^2) at retrieval time.
Approach
Learn multi-scale text probes (and video probes) via linear layers, slide them over the other modality's frame features computing cross-correlation (max over positions), then fuse all bidirectional scores with a graph-based cross-correlation reasoning module, avoiding cross-modal fusion and keeping O(n) complexity.
Key results
- On MSRVTT text-to-video with CLIP ViT-B/16, CLIP + BiHSSP reaches R@1 47.3 vs. 45.3 for CLIP finetune; STAN + BiHSSP reaches 50.8 R@1.
- On DiDeMo with CLIP ViT-B/32, CLIP + BiHSSP improves R@1 from 43.4 to 45.3, and the paper states it improves DiffusionRet by 1.2% on DiDeMo.
- Inference time on 20,000 video-text pairs drops by 26.5 seconds relative to X-Pool, and the paper reports 10.8 s vs 47.3 s in its timing figure.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add BiHSSP to CLIP | CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 43.1 -> 45.8 | No clear effect |
| Add BiHSSP to STAN | STAN with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 46.6 -> 48.1 | Helped |
| Add BiHSSP to CLIP | CLIP-ViT-B/32 · DiDeMo text-to-video | R@1: 43.4 -> 45.3 | Helped |
| Add BiHSSP to STAN | STAN with CLIP-ViT-B/32 · DiDeMo text-to-video | R@1: 46.2 -> 47.9 | Helped |
| Add BiHSSP to CLIP | CLIP-ViT-B/32 · MSVD text-to-video | R@1: 46.2 -> 47.9 | Helped |
| Add BiHSSP to STAN | STAN with CLIP-ViT-B/32 · MSVD text-to-video | R@1: 46.7 -> 48.1 | Helped |
| Add BiHSSP to STAN | STAN with CLIP-ViT-B/32 · ActivityNet text-to-video | R@1: 43.8 -> 45.8 | Helped |
| Add BiHSSP to CLIP | CLIP-ViT-B/32 · MSRVTT video-to-text | R@1: 43.1 -> 45.2 | Helped |
| Add BiHSSP to STAN | STAN with CLIP-ViT-B/32 · MSRVTT video-to-text | R@1: 46.6 -> 48.0 | Helped |
| Add BiHSSP to CLIPfinetune | CLIPfinetune · MSRVTT-1kA multi-caption retrieval | R@1: +4.2 | Helped |
| Use bidirectional rather than unidirectional sliding probe matching (ablation) | BiHSSP · MSRVTT text-to-video | R@1: 42.9 -> 44.6 | Helped |
| Add graph reasoning to bidirectional sliding probe matching (ablation) | BiHSSP · MSRVTT text-to-video | R@1: 44.6 -> 45.8 | Helped |
Surprising findings
- On ActivityNet, CLIP + BiHSSP R@1 (43.1) is lower than the CLIP finetune baseline (40.5)? No: BiHSSP improves ActivityNet R@1 over CLIP finetune but has higher MeanR (12.4 vs 7.4) than the baseline.
Limitations
- Efficiency comparison is shown only in one figure and uses a single GPU setting.
- Gains are mixed across datasets; on ActivityNet the MeanR of CLIP + BiHSSP is worse than the CLIP finetune baseline.
Experimental setup
Datasets: MSRVTT LSMDC DiDeMo ActivityNet Captions MSVD
Benchmarks: MSRVTT DiDeMo MSVD ActivityNet Captions
Metrics: R@1 R@5 R@10 MdR MeanR
Baselines: X-Pool TS2-Net PIDRo ProST DiffusionRet UATVR CLIP finetune STAN CLIP4Clip CenterCLIP
Builds on: CLIP CLIP4Clip STAN
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.87)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.87)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.87)
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval (NeurIPS 2025 · similarity 0.86)
- Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.