Research map / Vision-language and multimodal / CLIP and zero-shot
DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DGL, a cross-modal dynamic prompt-tuning method for text-video retrieval that generates text and frame prompts from a shared latent space and uses global-local video attention in frozen CLIP.
Problem
Adapting large pretrained image-text models like CLIP to text-video retrieval via fine-tuning is costly, and existing prompt-tuning methods fail to model cross-modal interaction and extract global video-level information.
Approach
Generate layer-wise local text and frame prompts jointly from a shared latent space (via a unified prompt transformer or unified linear projection), and add a global prompt with global-local video attention that shares parameters with the frozen CLIP visual encoder's attention, so the first global prompt serves as the video representation.
Key results
- With only 0.83M trainable parameters (ViT-B/32 and ViT-B/16), DGL-Linear achieves 44.7 T→V R@1 on MSR-VTT versus 43.1 for fully finetuned CLIP4Clip, and DGL-Transformer reaches 45.8 R@1 with 9.57M parameters.
- DGL-Linear with ViT-B/16 reaches 48.3 T→V R@1 on MSR-VTT, and 49.7 with QB-Norm, versus 45.6 for CLIP4Clip with ViT-B/16.
- On LSMDC, DGL-Linear achieves 21.4 T→V R@1 versus 20.7 for CLIP4Clip; on VATEX it reaches 56.2 vs 55.9.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add global and local video attention with shared parameters (ablation) | DGL-Transformer · MSR-VTT-9K, Text→Video | R@1: 43.5 -> 45.8 | Helped |
| Add global video attention (ablation) | DGL-Transformer with local attention · MSR-VTT-9K, Text→Video | R@1: 42.0 -> 43.5 | Helped |
| Share parameters between global and local attention (ablation) | DGL-Transformer with global and local attention · MSR-VTT-9K, Text→Video | R@1: 43.5 -> 45.8 | Helped |
| Use 4 prefix and 4 postfix text prompts (ablation) | DGL-Transformer · MSR-VTT-9K, Text→Video | R@1: 43.9 -> 45.8; 45.0 -> 45.8 | Helped |
| Project visual features to text prompts instead of projecting text to visual prompts (ablation) | DGL-Linear · MSR-VTT-9K, Text→Video | R@1: 43.4 -> 44.7 | Helped |
| Generate cross-modal prompts from a shared latent space rather than separately (ablation) | DGL-Transformer · MSR-VTT-9K, Text→Video | R@1: 43.8 -> 45.8 | Helped |
| Add QB-Norm | DGL-Transformer, CLIP ViT-B/32 · MSR-VTT-9K, Text→Video | R@1: 45.8 -> 47.0 | Helped |
| Add QB-Norm | DGL-Linear, CLIP ViT-B/16 · MSR-VTT-9K, Text→Video | R@1: 48.3 -> 49.7 | Helped |
| Add QB-Norm | DGL-Linear · VATEX, Text→Video | R@1: 56.2 -> 57.3 | Helped |
| Add QB-Norm | DGL-Linear · LSMDC, Text→Video | R@1: 21.4 -> 21.6 | Helped |
| Add QB-Norm | DGL-Linear · ActivityNet, Text→Video | R@1: 38.6 -> 43.1 | Helped |
| Use DGL-Linear instead of CLIP4Clip | CLIP ViT-B/32 · ActivityNet, Text→Video | R@1: 40.5 -> 38.6 | Hurt |
Surprising findings
- Visual-to-text linear projection outperforms the text-to-visual direction for DGL-Linear on text-to-video R@1 (44.7 vs 43.4).
Limitations
- Adapter baselines consume more GPU memory than fully fine-tuned CLIP4Clip, so the efficiency advantage is mostly in trainable parameters rather than memory for all variants.
- Results on the BLIP and X-CLIP comparisons are limited and partly use frozen backbones, so comparisons are not fully matched.
Experimental setup
Datasets: MSR-VTT VATEX LSMDC ActivityNet
Benchmarks: MSR-VTT VATEX LSMDC ActivityNet
Metrics: R@1 R@5 R@10 MnR trainable parameters GPU memory usage
Baselines: CLIP4Clip Efficient Prompt VPT UPT VoP Visual-Text Adapter Video-Text Adapter QB-Norm X-CLIP BLIP
Builds on: CLIP4Clip VPT UPT Efficient Prompt VoP
Closest papers by meaning
- ViLT-CLIP: Video and Language Tuning CLIP with Multimodal Prompt Learning and Scenario-Guided Optimization (AAAI 2024 · similarity 0.87)
- DiscoVLA: Discrepancy Reduction in Vision, Language, and Alignment for Parameter-Efficient Video-Text Retrieval (CVPR 2025 · similarity 0.87)
- Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis (CVPR 2025 · similarity 0.86)
- Re-thinking Temporal Search for Long-Form Video Understanding (CVPR 2025 · similarity 0.86)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.86)
- The Devil is in the Prompts: Retrieval-Augmented Prompt Optimization for Text-to-Video Generation (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.