Research map / Vision-language and multimodal / CLIP and zero-shot
Joint Scheduling of Causal Prompts and Tasks for Multi-Task Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes JSCPT, which combines causal prompt feature selection with a task-prompt scheduler and bi-level optimization to improve multi-task vision-language prompt tuning of CLIP.
Problem
Multi-task prompt learning for pre-trained VLMs suffers from spurious correlations in visual features and dynamic, conflicting task relationships, which degrade performance.
Approach
Learn task-shared and task-specific vision-language prompts with learnable causal masks (counterfactual feature intervention), and schedule task loss weights using a causal-effect score and a gradient-based inter-task affinity, optimized via bi-level optimization with implicit gradients (Neumann series approximation).
Key results
- On the Large-Scale MTL benchmark with 20 shots, JSCPT reaches 84.4% average accuracy, 1.4 points above the next best MVLPT-style baseline reported in the text.
- On Office-Home, JSCPT achieves 87.9% average accuracy at 20% split, outperforming MmAP (87.1%).
- On MiniDomainNet, JSCPT achieves 85.1% (1% split) and 86.4% (2% split), beating MmAP by 0.7 and 0.8 points.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply joint scheduling of causal prompts and tasks | CLIP ViT-B/16 multi-task prompt learning · Office-Home 10% and 20% Avg. | accuracy: 85.7±0.3 -> 86.5±0.3; 87.1±0.2 -> 87.9±0.3 | Helped |
| Apply joint scheduling of causal prompts and tasks | CLIP ViT-B/16 multi-task prompt learning · MiniDomainNet 1% and 2% Avg. | accuracy: 84.4±0.2 -> 85.1±0.2; 85.6±0.5 -> 86.4±0.3 | Helped |
| Apply joint scheduling of causal prompts and tasks | CLIP ViT-B/16 multi-task prompt learning · Large-Scale MTL benchmark 1-, 5-, 10-, and 20-shot | accuracy: 68.7 -> 70.1; 75.9 -> 77.2; 79.3 -> 80.8; 83.0 -> 84.4 | Helped |
| Add learnable feature selection instead of fixed masks (ablation) | JSCPT · Office-Home 10% Avg. | accuracy: 85.4 -> 86.5 | Helped |
| Add learnable feature selection instead of fixed masks (ablation) | JSCPT · Office-Home 20% Avg. | accuracy: 87.0 -> 87.9 | Helped |
| Add causal-effect-based prompt scheduling (ablation) | JSCPT · Office-Home 10% Avg. | accuracy: 86.1 -> 86.5 | Helped |
| Add causal-effect-based prompt scheduling (ablation) | JSCPT · Office-Home 20% Avg. | accuracy: 87.3 -> 87.9 | Helped |
| Add inter-task-affinity-based task scheduling (ablation) | JSCPT · Office-Home 10% Avg. | accuracy: 85.9 -> 86.5 | Helped |
| Add inter-task-affinity-based task scheduling (ablation) | JSCPT · Office-Home 20% Avg. | accuracy: 87.2 -> 87.9 | Helped |
| Replace task scheduling with CAGrad (ablation) | JSCPT · Office-Home 10% Avg. | accuracy: 85.6 -> 86.5 | Hurt |
| Replace task scheduling with CAGrad (ablation) | JSCPT · Office-Home 20% Avg. | accuracy: 87.1 -> 87.9 | Hurt |
Surprising findings
- Removing feature selection causes the largest ablation drop (86.5 to 85.4 at 10%), larger than removing task scheduling.
Limitations
- Gains over the strongest baseline (MmAP) are modest, under 1 point on most splits.
- Scheduler adds a bi-level optimization with a separate developing set sampled from validation data, increasing complexity.
Experimental setup
Datasets: Office-Home MiniDomainNet Large-scale MTL benchmark (10 visual datasets, including Food101, OxfordFlowers, OxfordPets, StanfordCars, SUN397)
Benchmarks: Office-Home MiniDomainNet Large-scale MTL benchmark
Metrics: accuracy (%)
Baselines: Zero-Shot CLIP CoOp VPT MaPLe CLIP-Adapter MVLPT MmAP
Builds on: CoOp VPT MaPLe MVLPT MmAP CAGrad Granger causal objective
Closest papers by meaning
- Weighted Multi-Prompt Learning with Description-free Large Language Model Distillation (ICLR 2025 · similarity 0.89)
- MmAP: Multi-Modal Alignment Prompt for Cross-Domain Multi-Task Learning (AAAI 2024 · similarity 0.89)
- Task-Aware Clustering for Prompting Vision-Language Models (CVPR 2025 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models (CVPR 2025 · similarity 0.89)
- Make Prompts Adaptable: Bayesian Modeling for Vision-Language Prompt Learning with Data-Dependent Prior (AAAI 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.