Research map / Video / Video-language understanding
Diffusion-Inspired Truncated Sampler for Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DITS, a diffusion-inspired truncated sampler that starts from the text embedding and runs a short contrastive-trained diffusion flow toward the video embedding to close the text-video modality gap in CLIP-based retrieval.
Problem
Text-video retrieval suffers from a modality gap between text and video embeddings that persists in the joint space; vanilla diffusion alignment with L2 loss and isotropic Gaussian starting points does not suit the ranking nature of retrieval.
Approach
Replace the isotropic Gaussian starting point of diffusion with the text embedding, define a truncated (few-step) reverse flow from text to video, and train the alignment network jointly with the contrastive (symmetric cross-entropy) loss instead of an L2 denoising loss.
Key results
- DITS with CLIP ViT-B/16 reaches 55.0 R@1 on MSRVTT (vs T-MASS 52.7) and 55.8 R@1 on DiDeMo (vs T-MASS 53.3).
- DITS with CLIP ViT-B/32 reaches 51.9 R@1 on MSRVTT, outperforming X-Pool baseline's 46.9.
- On Charades, DITS improves R@1 over X-Pool by 4.3 (ViT-B/32) and 6.7 (ViT-B/16) points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| apply DITS instead of T-MASS | CLIP-ViT-B/32 · MSRVTT Retrieval | R@1: 50.2 -> 51.9 | Helped |
| apply DITS instead of T-MASS | CLIP-ViT-B/32 · LSMDC Retrieval | R@1: 28.9 -> 28.2 | Hurt |
| apply DITS instead of T-MASS | CLIP-ViT-B/16 · LSMDC Retrieval | R@1: 30.3 -> 31.0 | Helped |
| replace DITS with a fixed prior (σ²=1.0) (ablation) | X-Pool on MSRVTT-1k · MSRVTT-1k | R@1: 46.9 -> 28.0 | Hurt |
| replace DITS with a fixed prior (σ²=0.1) (ablation) | X-Pool on MSRVTT-1k · MSRVTT-1k | R@1: 46.9 -> 45.1 | Hurt |
| use pretrained diffusion alignment with L2 and Lsce losses, without truncation (ablation) | X-Pool on MSRVTT-1k · MSRVTT-1k | R@1: 46.9 -> 35.4 | Hurt |
| fine-tune diffusion alignment with Lsce loss, without truncation (ablation) | X-Pool on MSRVTT-1k · MSRVTT-1k | R@1: 46.9 -> 46.5 | No clear effect |
| set truncated timestamps T′=1 instead of T′=10 (ablation) | DITS on MSRVTT-1k · MSRVTT-1k | R@1: 51.9 -> 49.2 | Hurt |
| set truncated timestamps T′=20 instead of T′=10 (ablation) | DITS on MSRVTT-1k · MSRVTT-1k | R@1: 51.9 -> 47.0 | Hurt |
| freeze CLIP during DITS alignment rather than jointly train it (ablation) | DITS on MSRVTT-1k · MSRVTT-1k | R@1: 51.9 -> 39.2 | Hurt |
| add text condition to the alignment network (ablation) | DITS on MSRVTT-1k · MSRVTT-1k | R@1: 50.1 -> 42.3 | Hurt |
| add video condition to the alignment network (ablation) | DITS on MSRVTT-1k · MSRVTT-1k | R@1: 50.1 -> 23.1 | Hurt |
Surprising findings
- Conditioning the denoising network on text or video embeddings hurts performance; the unconditional alignment network works best.
- The vanilla diffusion pretraining with L2 loss degrades retrieval (R@1 35.4 vs 46.9 baseline), and the L1 variant is worse still.
Limitations
- Relies on large amounts of labeled text-video pairs; zero-shot and few-shot settings are unexplored.
- Robustness to noisy or incomplete text and video data is not evaluated.
- Optimal truncation step count T' is chosen empirically and performance drops sharply for large T' (e.g., 0.2 R@1 at T'=40).
Experimental setup
Datasets: MSRVTT LSMDC DiDeMo VATEX Charades
Benchmarks: MSRVTT LSMDC DiDeMo VATEX Charades
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: X-Pool STAN ProST DiffusionRet UATVR UCOFIA TEFAL CLIP-ViP T-MASS
Builds on: X-Pool DDPM DiT CLIP DiffusionRet
Closest papers by meaning
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval (CVPR 2024 · similarity 0.88)
- Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search (NeurIPS 2025 · similarity 0.88)
- Divot: Diffusion Powers Video Tokenizer for Comprehension and Generation (CVPR 2025 · similarity 0.87)
- Bridging Information Asymmetry in Text-video Retrieval: A Data-centric Approach (ICLR 2025 · similarity 0.87)
- Rebalancing Contrastive Alignment with Bottlenecked Semantic Increments in Text-Video Retrieval (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.