Research map / Vision-language and multimodal / CLIP and zero-shot
MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MmAP, a Kronecker-product multi-modal prompt for CLIP, combined with gradient-similarity task grouping and group-shared plus task-specific prompts for parameter-efficient cross-domain multi-task image classification.
Problem
Adapting CLIP to multi-task image recognition parameter-efficiently, where existing prompt methods tune only one modality and multi-task full fine-tuning needs a separate decoder/parameters per task.
Approach
A Multi-modal Alignment Prompt generates text and visual prompts from one shared source prompt via Kronecker products with two small scaling matrices, keeping the modalities aligned; tasks are grouped by gradient similarity, with a group-shared MmAP plus a task-specific MmAP per task.
Key results
- On Office-Home with 10% training data, the method reaches 86.5% average accuracy versus 86.1% for multi-task full fine-tuning, using 0.13M trainable parameters (about 0.09% of CLIP).
- On MiniDomainNet, the method achieves 84.9% (1% split) and 86.1% (2% split), the best among compared methods.
- Versus MaPLe-MT, the method gains about 1 point average on Office-Home; the MmAP alone is 0.54% above an MLP-based prompt generator with 0.13M vs 3.96M parameters.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use MmAP instead of multi-task full fine-tuning | CLIP ViT-B/16 · Office-Home 10% average | accuracy (%): 86.1$\pm$1.1 -> 86.5$\pm$0.6 | Helped |
| use MmAP instead of multi-task full fine-tuning | CLIP ViT-B/16 · Office-Home 20% average | accuracy (%): 87.8$\pm$0.8 -> 87.8$\pm$0.5 | No clear effect |
| use MmAP instead of MaPLe-MT | CLIP ViT-B/16 · MiniDomainNet 1% average | accuracy (%): 84.1$\pm$0.3 -> 84.9$\pm$0.4 | Helped |
| use MmAP instead of MaPLe-MT | CLIP ViT-B/16 · MiniDomainNet 2% average | accuracy (%): 85.1$\pm$0.3 -> 86.1$\pm$0.2 | Helped |
| replace joint text-and-visual prompt training with two MLP layers that generate both prompts (ablation) | CLIP prompt tuning · Office-Home four tasks | accuracy (%): +0.5% | Helped |
| replace the two-MLP prompt generator with MmAP using a Kronecker Product (ablation) | CLIP prompt tuning · Office-Home four tasks | accuracy (%): +0.54% | Helped |
| replace random task grouping with gradient-based task grouping (ablation) | MmAP multi-task prompt framework · Office-Home 10% | accuracy (%): +0.68% | Helped |
| replace random task grouping with gradient-based task grouping (ablation) | MmAP multi-task prompt framework · Office-Home 20% | accuracy (%): +0.85% | Helped |
| replace a single group containing all tasks with gradient-based task grouping (ablation) | MmAP multi-task prompt framework · Office-Home 10% | accuracy (%): +0.39% | Helped |
| replace a single group containing all tasks with gradient-based task grouping (ablation) | MmAP multi-task prompt framework · Office-Home 20% | accuracy (%): +0.41% | Helped |
| add group-shared MmAP and gradient-based task grouping to task-specific MmAP (ablation) | MmAP multi-task prompt framework · Office-Home 10% | accuracy (%): 85.76 -> 86.48 | Helped |
| add group-shared MmAP and gradient-based task grouping to task-specific MmAP (ablation) | MmAP multi-task prompt framework · Office-Home 20% | accuracy (%): 86.97 -> 87.77 | Helped |
Surprising findings
- Full fine-tuning underperforms many parameter-efficient methods on MiniDomainNet, attributed to overfitting with limited data.
- All prompt methods fall below zero-shot CLIP in the 1-shot setting on some Office-Home tasks.
Limitations
- Gains over full fine-tuning are reported on a small data regime (limited shots per class), not full-data training.
- Experiments are limited to two cross-domain image classification datasets with CLIP ViT-B/16 only.
- Task grouping relies on gradient similarity snapshots, and the paper does not detail the group-search procedure for large task counts.
Experimental setup
Datasets: Office-Home MiniDomainNet
Benchmarks: Office-Home MiniDomainNet
Metrics: average accuracy (%) trainable parameters
Baselines: Zero-shot CLIP Full fine-tuning CoOp-MT VPT-MT MaPLe-MT CLIP-Adapter BitFit CoOp VPT
Builds on: CLIP CoOp VPT MaPLe
Closest papers by meaning
- LAMM: Label Alignment for Multi-Modal Prompt Learning (AAAI 2024 · similarity 0.89)
- Amend to Alignment: Decoupled Prompt Tuning for Mitigating Spurious Correlation in Vision-Language Models (ICML 2024 · similarity 0.89)
- Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning (CVPR 2025 · similarity 0.89)
- COMMA: Co-articulated Multi-Modal Learning (AAAI 2024 · similarity 0.89)
- VaMP: Variational Multi-Modal Prompt Learning for Vision-Language Models (NeurIPS 2025 · similarity 0.88)
- Vision-Language Models Create Cross-Modal Task Representations (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.