Research map / Video / Video-language understanding
GMMFormer: Gaussian-Mixture-Model Based Transformer for Efficient Partially Relevant Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes GMMFormer, a Gaussian-mixture-constrained Transformer that models clip representations implicitly for partially relevant video retrieval, plus a query diverse loss that separates queries describing the same video.
Problem
Partially relevant video retrieval (PRVR) requires clip modeling, but scanning-based clip construction is information-redundant and storage-heavy, and global frame interactions make frame embeddings similar so different moments get confused.
Approach
Replace explicit multi-scale sliding-window clip construction with Transformer frame interactions constrained by multi-scale Gaussian windows (a Gaussian mixture), so each frame attends to its neighbors and outputs contain multi-scale clip information in a length-M sequence; add a query diverse loss that pushes apart queries describing the same video.
Key results
- On TVR, GMMFormer reaches SumR 176.6 vs. 172.4 for prior SOTA MS-SL; on ActivityNet Captions 146.0 vs. 140.1; on Charades-STA 72.9 vs. 68.4.
- Compared with MS-SL, GMMFormer is reported as about 2.5x faster at retrieval and its memory overhead is about 20x smaller (e.g., 2.53M vs. 50.02M at database size 500).
- Ablation on TVR: GMMFormer block alone gives SumR 172.9 and query diverse loss alone 169.9, versus 163.5 for the baseline and 176.6 for the full model.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMMFormer blocks instead of vanilla Transformer blocks (ablation) | GMMFormer · TVR | SumR: 163.5 -> 172.9 | Helped |
| Add query diverse loss (ablation) | GMMFormer · TVR | SumR: 163.5 -> 169.9 | Helped |
| Remove query diverse loss (ablation) | GMMFormer · TVR | SumR: 176.6 -> 172.9 | Hurt |
| Replace Gaussian constraint window with Boxcar window (ablation) | GMMFormer · TVR | SumR: 176.6 -> 172.1 | Hurt |
| Replace Gaussian constraint window with Bartlett window (ablation) | GMMFormer · TVR | SumR: 176.6 -> 174.0 | Hurt |
| Remove the low-variance Gaussian block (ablation) | GMMFormer · ActivityNet Captions, M/V 0.00-0.25 | SumR | Hurt |
| Remove the medium-variance Gaussian block (ablation) | GMMFormer · ActivityNet Captions, M/V 0.25-0.50 | SumR | Hurt |
| Remove the high-variance Gaussian block (ablation) | GMMFormer · ActivityNet Captions, M/V 0.50-1.00 | SumR | Hurt |
| Remove the infinite-variance Gaussian block (ablation) | GMMFormer · ActivityNet Captions, M/V 0.50-1.00 | SumR | Hurt |
| Use compact implicit clip embeddings instead of scanning-based explicit clip construction | GMMFormer versus MS-SL · TVR, database size 2,500 | runtime (ms): 12.93 -> 4.56 | Helped |
| Use compact implicit clip embeddings instead of scanning-based explicit clip construction | GMMFormer versus MS-SL · TVR, database size 2,500 | memory usage (M): 250.11 -> 12.67 | Helped |
Limitations
- GMMFormer has more parameters (12.85M) and FLOPs (1.95G) than MS-SL (4.85M, 1.29G) due to parallel Gaussian blocks.
- Gaussian variances and loss weights are hand-set hyperparameters per dataset.
- Evaluated only on three video datasets with pre-extracted features.
Experimental setup
Datasets: TVR ActivityNet Captions Charades-STA
Benchmarks: TVR ActivityNet Captions Charades-STA
Metrics: R@1 R@5 R@10 R@100 SumR FLOPs parameters runtime (ms) memory usage (M)
Baselines: MS-SL CONQUER XML ReLoCLNet CLIP4Clip Cap4Video HGR DE++ RIVRL HTM CE W2VV
Builds on: MS-SL Transformer triplet ranking loss infoNCE
Closest papers by meaning
- Fewer Steps, Better Performance: Efficient Cross-Modal Clip Trimming for Video Moment Retrieval Using Language (AAAI 2024 · similarity 0.87)
- Bridging the Semantic Granularity Gap Between Text and Frame Representations for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.87)
- Generative Video Diffusion for Unseen Novel Semantic Video Moment Retrieval (AAAI 2025 · similarity 0.86)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.86)
- Ambiguity-Restrained Text-Video Representation Learning for Partially Relevant Video Retrieval (AAAI 2025 · similarity 0.85)
- Composed Video Retrieval via Enriched Context and Discriminative Embeddings (CVPR 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.