Research map / Vision-language and multimodal / CLIP and zero-shot
Task-Oriented Multi-Modal Mutual Leaning for Vision-Language Models
Sifan Long, Zhen Zhao, Junkun Yuan, Zichang Tan, Jiangjiang Liu, Luping Zhou, Shengsheng Wang, Jingdong Wang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add class-aware text prompts (CTP) to the text branch (ablation) | CoOp baseline · Base-to-new, 11-dataset average | harmonic mean accuracy (%): 71.66 -> 76.64 | Improved |
| add text-guided feature tuning (TFT) to the image branch (ablation) | CoOp baseline · Base-to-new, 11-dataset average | harmonic mean accuracy (%): 71.66 -> 77.24 | Improved |
| combine CTP and TFT (ablation) | CoOp baseline · Base-to-new, 11-dataset average | harmonic mean accuracy (%): 71.66 -> 79.02 | Improved |
| use text-image attention (CTP) instead of Linear-ReLU-Linear block for prompt learning (ablation) | text branch · Base-to-new, 11-dataset average | harmonic mean accuracy (%): 75.83 -> 76.64 | Improved |
| use text-image attention (TFT) instead of Linear-ReLU-Linear block for feature tuning (ablation) | image branch · Base-to-new, 11-dataset average | harmonic mean accuracy (%): 75.94 -> 77.24 | Improved |
| inject image information into prompts of all classes identically (CoCoOp) instead of class-aware prompts | CoOp · Base-to-new, base classes, 11-dataset average | accuracy (%): 82.69 -> 80.47 | Hurt |
| replace CoCoOp with CTP + TFT | CoCoOp · Base-to-new, new classes, 11-dataset average | accuracy (%): 71.69 -> 75.72 | Improved |
| replace CoCoOp with CTP + TFT in 16-shot learning | CoCoOp · 16-shot, 11-dataset average | accuracy (%): 74.79 -> 80.32 | Improved |
| replace CoCoOp with CTP + TFT in cross-dataset transfer | CoCoOp · Cross-dataset transfer, target average | accuracy (%): 65.74 -> 66.47 | Improved |
| vary the balance weight lambda (ablation) | Ours · Base-to-new | accuracy | No clear effect |
Closest papers by meaning
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.91)
- Concept-Guided Prompt Learning for Generalization in Vision-Language Models (AAAI 2024 · similarity 0.90)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.90)
- CoPL: Contextual Prompt Learning for Vision-Language Understanding (AAAI 2024 · similarity 0.90)
- Compound Text-Guided Prompt Tuning via Image-Adaptive Cues (AAAI 2024 · similarity 0.90)
- ProAPO: Progressively Automatic Prompt Optimization for Visual Classification (CVPR 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
