Research map / Video / Video-language understanding
TempMe: Video Temporal Token Merging for Efficient Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes TempMe, a parameter- and inference-efficient text-video retrieval method that progressively merges temporally redundant tokens across adjacent clips on top of a LoRA-adapted CLIP.
Problem
Text-video retrieval with image-pretrained CLIP processes each sampled frame independently, producing many tokens; existing parameter-efficient methods still incur high inference cost, and image token compression ignores temporal redundancy across frames.
Approach
A progressive multi-granularity framework: ImgMe blocks merge spatial tokens within each frame via ToMe-style bipartite matching, then ClipMe blocks progressively concatenate adjacent clips and merge tokens across frames (cross-clip then intra-clip merging) with trainable clip positional embeddings, so the video is reduced to a single clip while temporal modeling happens in later layers.
Key results
- On MSRVTT with CLIP-ViT-B/16 (LoRA, 0.50M params), TempMe reaches 49.0 R@1 and 206.7 R-Sum at 121.4 GFLOPs, versus VoP at 202.3 R-Sum and 246.2 GFLOPs.
- TempMe reduces output tokens to 127 (5%) versus 2364 for LoRA on CLIP-ViT-B/16 and achieves 45.1 videos/s throughput.
- With full fine-tuning on CLIP-ViT-B/16 on MSRVTT, TempMe improves R-Sum by 7.9% over CLIP4Clip (210.2 vs 202.3 in its table, 1.57x training speedup, 75.2% training memory).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use TempMe instead of the strongest listed parameter-efficient baseline | MSRVTT text-to-video with CLIP-ViT-B/32 · MSRVTT | R-Sum: 194.8 -> 198.6 | Helped |
| Use TempMe instead of RAP | MSRVTT text-to-video with CLIP-ViT-B/16 · MSRVTT | R-Sum: 202.4 -> 206.7 | Helped |
| Add ImgMe to the LoRA baseline (ablation) | TempMe on MSRVTT · MSRVTT | R-Sum: 193.0 -> 191.4 | Hurt |
| Add ClipMe to LoRA + ImgMe (ablation) | TempMe on MSRVTT · MSRVTT | R-Sum: 191.4 -> 198.6 | Helped |
| Add temporal modeling without token reduction (ablation) | LoRA on MSRVTT · MSRVTT | R-Sum: 193.0 -> 199.7 | Helped |
| Add token reduction without temporal modeling (ablation) | LoRA on MSRVTT · MSRVTT | R-Sum: 193.0 -> 188.8 | Hurt |
| Replace progressive merging with one-shot merging (ablation) | TempMe on MSRVTT · MSRVTT | R@1: 46.1 -> 44.7 | Hurt |
| Stop merging before all frames form one clip (ablation) | TempMe on MSRVTT · MSRVTT | R@1: 46.1 -> 43.6 | Hurt |
| Start clip merging earlier (ablation) | TempMe on MSRVTT · MSRVTT | R@1: 46.1 -> 40.8 | Hurt |
| Add clip positional embeddings (ablation) | TempMe on MSRVTT · MSRVTT | R-Sum: 197.6 -> 198.6 | Helped |
| Reduce sampled frames from 12 to 8 (ablation) | TempMe on MSRVTT · MSRVTT | R-Sum: 198.6 -> 195.6 | Hurt |
| Add TempMe to full fine-tuning | CLIP4Clip on MSRVTT · MSRVTT | R-Sum: 202.3 -> 210.2 | Helped |
Surprising findings
- Token reduction alone (without temporal modeling) causes a 4.2% R-Sum drop, indicating that temporal modeling is needed before merging tokens across frames.
- Merging all frames into a single clip at once increases complexity due to quadratic attention cost; progressive merging performs better at lower cost.
Limitations
- Only accelerates the video encoder; the text encoder is not compressed.
- Transfer to video foundation models is feasible only when the video encoder can be treated as an image encoder (e.g., UMT); incompatible with CLIP-ViP-style joint token processing.
- Frame reduction to 8 frames costs about 1.4% R-Sum, so performance depends on the number of sampled frames.
Experimental setup
Datasets: MSRVTT ActivityNet DiDeMo LSMDC MSR-QA
Benchmarks: MSRVTT ActivityNet DiDeMo LSMDC MSR-QA
Metrics: R@1 R@5 R@10 R-Sum MnR GFLOPs throughput (videos/s) training speed GPU memory
Baselines: CLIP4Clip Prompt Adapter LoRA PLEVU VoP DGL MV-Adapter RAP EVIT DiffRate ToMe
Builds on: ToMe LoRA CLIP4Clip VoP DiffRate
Closest papers by meaning
- Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learning (AAAI 2024 · similarity 0.89)
- Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrieval (CVPR 2025 · similarity 0.88)
- Mitigating Semantic Collapse in Partially Relevant Video Retrieval (NeurIPS 2025 · similarity 0.88)
- Text Proxy: Decomposing Retrieval from a 1-to-N Relationship into N 1-to-1 Relationships for Text-Video Retrieval (AAAI 2025 · similarity 0.88)
- From Play to Replay: Composed Video Retrieval for Temporally Fine-Grained Videos (NeurIPS 2025 · similarity 0.87)
- HoliTom: Holistic Token Merging for Fast Video Large Language Models (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.