Research map / Video / Video-language understanding
Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Video-ColBERT, a bi-encoder for text-to-video retrieval that applies ColBERT-style mean-MaxSim late interaction over both frame-level and temporally contextualized video features, trained with a dual sigmoid loss.
Problem
Text-to-video retrieval with bi-encoders is limited when single-vector representations fail to capture fine-grained query concepts and video content.
Approach
Adapts ColBERT late interaction to video by replacing MaxSim summation with a mean (MeanMaxSim), applying it at both spatial (frame) and spatio-temporal (contextualized) levels, adding query and visual expansion tokens, and training each interaction with a separate sigmoid loss.
Key results
- On MSR-VTT, Video-ColBERT with SigLIP-B/16 reaches R@1 51.5 and with CLIP-B/32 reaches R@1 48.1, exceeding bi-encoder baselines such as DRL (47.4) at comparable size.
- With SigLIP-B/16 Video-ColBERT sets new bi-encoder state-of-the-art on MSR-VTT, MSVD (R@1 55.2) and VATEX (R@1 68.0) among compared methods.
- Combining frame and video MMS scores (48.1 R@1) outperforms frame-only (44.3) and video-only (47.0) interactions, and the sigmoid loss beats InfoNCE (48.1 vs 45.3 R@1 under dual loss).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| combine frame-level and video-level MMS scores instead of using frame-level MMS alone (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 44.3 -> 48.1 | Helped |
| combine frame-level and video-level MMS scores instead of using video-level MMS alone (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 47.0 -> 48.1 | Helped |
| use reciprocal rank fusion instead of summing frame and video MMS scores (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 48.1 -> 46.8 | Hurt |
| use dual sigmoid loss instead of combined InfoNCE loss (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 45.1 -> 48.1 | Helped |
| use dual sigmoid loss instead of combined sigmoid loss (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 47.1 -> 48.1 | Helped |
| use dual sigmoid loss instead of dual InfoNCE loss (ablation) | VIDEO-COLBERT · MSR-VTT | R@1: 45.3 -> 48.1 | Helped |
| add pad-token query augmentation (ablation) | VIDEO-COLBERT with CLIP-B/32 · MSR-VTT | R@1: 45.3 -> 48.1 | Helped |
| add pad-token query augmentation (ablation) | VIDEO-COLBERT with SigLIP-B/16 · MSR-VTT | R@1: 51.0 -> 51.5 | Helped |
| increase sampled video frames from 4 to 12 (ablation) | VIDEO-COLBERT with CLIP-B/32 · MSR-VTT | R@1: 45.2 -> 48.1 | Helped |
| increase sampled video frames from 12 to 20 (ablation) | VIDEO-COLBERT with CLIP-B/32 · MSR-VTT | R@1: 48.1 -> 48.4 | No clear effect |
| increase temporal transformer depth from 2 to 4 layers (ablation) | VIDEO-COLBERT with CLIP-B/32 · MSR-VTT | R@1: 47.0 -> 48.1 | Helped |
| increase temporal transformer depth from 4 to 8 layers (ablation) | VIDEO-COLBERT with CLIP-B/32 · MSR-VTT | R@1: 48.1 -> 48.5 | No clear effect |
Surprising findings
- Reciprocal rank fusion of frame and video scores hurt retrieval compared with simply summing the MMS scores.
- The SigLIP-based variant performs relatively poorly on ActivityNet, attributed to SigLIP's 16-token text pretraining length limit.
Limitations
- SigLIP variant underperforms on long-caption ActivityNet paragraphs due to text length limits.
- Query augmentation benefits short queries but hurts longer ones (>20 tokens).
Experimental setup
Datasets: MSR-VTT MSVD VATEX DiDeMo ActivityNet
Benchmarks: MSR-VTT MSVD VATEX DiDeMo ActivityNet
Metrics: R@1 R@5 R@10 nDCG@10
Baselines: CLIP4Clip CLIP2TV TS2-Net X-CLIP DRL CenterCLIP Cap4Video X-Pool InternVideo2-6B SigLIP4CLIP
Builds on: ColBERT CLIP4Clip SigLIP FILIP DRL
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.88)
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.88)
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.88)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.87)
- Learning Dynamic Similarity by Bidirectional Hierarchical Sliding Semantic Probe for Efficient Text Video Retrieval (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.