Research map / Vision-language and multimodal / CLIP and zero-shot
LLM-enhanced Action-aware Multi-modal Prompt Tuning for Image-Text Matching
Mengxiao Tian, Xinxiao Wu, Shuo Yang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add action-aware prompts from LLM knowledge versus plain learnable visual prompts (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 55.3 -> 58.4 | Improved |
| add action state prompt to action triplet prompt (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 56.2 -> 58.1 | Improved |
| use LLM-generated action triplet prompt instead of hand-crafted template (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 55.5 -> 56.2 | Improved |
| add visual prompt (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 58.1 -> 58.4 | Improved |
| add attribute knowledge to action triplets (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 58.2 -> 58.4 | Improved |
| use adaptive interaction module instead of concatenation (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 57.4 -> 58.4 | Improved |
| use two-stage training instead of one-stage (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 56.8 -> 58.4 | Improved |
| use two-stage training instead of combined losses (ablation) | CLIP ViT-B/16 · COCO 5K | image-to-text R@1: 56.5 -> 58.4 | Improved |
| swap GPT-3.5 for smaller open LLM (Vicuna-7B) (ablation) | action knowledge generation · COCO 5K | image-to-text R@1: 58.4 -> 58.4 | No clear effect |
| apply method to CLIP ViT-L/14-336 | CLIP · COCO 5K | Rsum: 394.2 -> 438.5 | Mixed |
| apply method to CLIP ViT-B/32 | CLIP · Flickr30K 1K | Rsum: 502.3 -> 535.6 | Improved |
Closest papers by meaning
- Kronecker Mask and Interpretive Prompts are Language-Action Video Learners (ICLR 2025 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.89)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.89)
- Frozen Transformers in Language Models Are Effective Visual Encoder Layers (ICLR 2024 · similarity 0.88)
- LION: Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge (CVPR 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
