Research map / Video / Video-language understanding
Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes T-MASS, which models text as a stochastic embedding ("text mass") with a similarity-aware radius and support text regularization, improving CLIP-based text-video retrieval over X-Pool on five benchmarks.
Problem
Single-point text embeddings are not expressive enough to capture the redundant semantics of videos, making text-video alignment in a joint embedding space difficult.
Approach
Represent text as a stochastic embedding ball (text mass) via reparameterization, with a learnable similarity-aware radius that scales the mass per text-video pair, a stochastic symmetric contrastive loss, a support text vector regularizer, and inference that picks the sampled text embedding closest to the video.
Key results
- T-MASS (ViT-B/32) reaches 50.2 R@1 on MSRVTT, versus 46.9 for the X-Pool baseline.
- T-MASS improves X-Pool by 6.0 R@1 on DiDeMo with ViT-B/16 (53.3 vs 47.3).
- On Charades with ViT-B/16, T-MASS reaches 26.7 R@1 versus 20.7 for X-Pool.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| model text as stochastic embeddings with a similarity-aware radius and support text regularization | X-Pool with CLIP-ViT-B/32 · LSMDC text-to-video | R@1: 26.8 -> 28.9 | Helped |
| model text as stochastic embeddings with a similarity-aware radius and support text regularization | X-Pool with CLIP-ViT-B/16 · DiDeMo text-to-video | R@1: 50.5 -> 53.3 | Helped |
| model text as stochastic embeddings with a similarity-aware radius and support text regularization | X-Pool with CLIP-ViT-B/16 · MSRVTT text-to-video | R@1: 54.2 -> 52.7 | Hurt |
| add a similarity-aware radius module (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 46.9 -> 49.1 | Helped |
| replace the fixed similarity radius with a learned scalar radius (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 48.7 -> 49.2 | Helped |
| use a learned linear-layer radius instead of a learned scalar radius (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 49.2 -> 49.1 | No clear effect |
| add stochastic text embedding ts and stochastic loss Ls while retaining Lce (ablation) | X-Pool with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 46.9 -> 48.5 | Helped |
| remove Lce and train with stochastic text embedding ts and Ls (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 48.5 -> 49.1 | Helped |
| add support text regularization Lsup (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT text-to-video | R@1: 49.1 -> 50.2 | Helped |
| use stochastic sampling at inference instead of the original text embedding (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT-1K text-to-video | R@1: 44.4 -> 50.2 | Helped |
| increase inference sampling trials from 5 to 20 (ablation) | T-MASS with CLIP-ViT-B/32 · MSRVTT-1K text-to-video | R@1: 46.8 -> 50.2 | Helped |
Surprising findings
- Regularizing the deterministic text embedding t with cross-entropy loss harms performance once stochastic text embeddings are used.
- A learnable scalar radius performs nearly as well as the linear-layer radius, suggesting the method is not sensitive to the radius network design.
Limitations
- On MSRVTT with ViT-B/16, CLIP-ViP (which uses extra post-pretraining data) outperforms T-MASS (54.2 vs 52.7 R@1).
- Inference requires sampling 20 stochastic text embeddings per query, adding computational cost.
- Experiments use only 12 frames at 224x224 and batch size 32, while some baselines use larger settings.
Experimental setup
Datasets: MSRVTT LSMDC DiDeMo VATEX Charades
Benchmarks: MSRVTT LSMDC DiDeMo VATEX Charades
Metrics: R@1 R@5 R@10 MdR MnR
Baselines: X-Pool DiffusionRet UATVR TEFAL CLIP-ViP CLIP4Clip CenterCLIP TS2-Net ClipBERT
Builds on: X-Pool CLIP symmetric cross-entropy loss reparameterization trick
Closest papers by meaning
- Diffusion-Inspired Truncated Sampler for Text-Video Retrieval (NeurIPS 2024 · similarity 0.88)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.87)
- TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval (ICLR 2025 · similarity 0.87)
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.87)
- MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction (ICLR 2026 · similarity 0.87)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.