Research map / Architectures and efficiency / Pruning and efficient tuning
pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes pMoE, a mixture-of-experts visual prompt tuning method that prompts frozen ViTs from several pre-trained experts (e.g., DINO, MoCo v3, MAE, CLIP, LVM-Med) using expert-specific prompt tokens and a learnable dispatcher that fuses them layer by layer.
Problem
Visual prompt tuning typically draws knowledge from a single pre-trained model, missing potential synergies from integrating diverse general and medical domain experts during adaptation.
Approach
Insert expert-specific prompt tokens for each frozen pre-trained ViT expert and a shared learnable dispatcher that, at each prompt layer, computes softmax weights over all experts' prompt tokens to form integrated prompt tokens for each expert, enabling cross-expert communication while keeping expert-specific patch processing.
Key results
- On VTAB-1K with ImageNet-21K ViT-B/16, LSPT + pMoE reaches 80.31 average vs. 77.95 for LSPT.
- On FGVC with DINO v2 ViT-B, LSPT + pMoE improves CUB to 86.07 vs. 84.85 for LSPT (1.10X params).
- On ADE20K segmentation with MAE, LSPT + pMoE reaches 40.68 mIoU (SS) vs. 39.72 for LSPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add pMoE to LSPT | LSPT on VTAB-1K · VTAB-1K Average | accuracy: 77.95 -> 80.31 | Helped |
| Add pMoE to LSPT | LSPT on DINOv2 ViT-B/16 · FGVC (CUB, Flowers, Cars, Dogs, NABirds) | accuracy: CUB: 84.85 -> 86.07; Flowers: 95.57 -> 96.58; Cars: 80.63 -> 81.12; Dogs: 84.53 -> 85.17; NABirds: 77.52 -> 78.06 | Helped |
| Add pMoE to LSPT | LSPT on medical classification · Medical color, X-ray, OCT/CT/MRI datasets | accuracy | Helped |
| Add pMoE to LSPT | LSPT on ADE20K · ADE20K, MAE ViT-B/16 | mIoU: 39.72 -> 40.68 (SS); 41.51 -> 42.87 (MS) | Helped |
| Add pMoE to LSPT | LSPT on medical segmentation · Kvasir-seg and Skin lesion | mIoU: Kvasir-seg (MAE): 45.81 -> 47.95; Skin (MAE): 77.63 -> 80.35 | Helped |
| Add Expert Prompt Tokens (ablation) | pMoE without EPTs or dispatcher · CUB | accuracy: 82.95 -> 83.07 | Helped |
| Add dispatcher (ablation) | pMoE with EPTs · CUB | accuracy: 83.07 -> 83.38 | Helped |
| Use DINO and CLIP experts rather than MoCo v3 and CLIP (ablation) | pMoE · CUB | accuracy: 83.38 -> 83.76 | Helped |
| Increase dispatcher experts from 3 to 6 (ablation) | pMoE · CUB | accuracy: 83.17 -> 83.38 | Helped |
| Increase dispatcher experts from 6 to 9 (ablation) | pMoE · CUB | accuracy: 83.38 -> 83.43 | Helped |
| Increase prompt layers from 3 to 12 (ablation) | pMoE · CUB | accuracy: 83.08 -> 83.38 | Helped |
| Increase prompt layers from 3 to 12 (ablation) | pMoE · NABirds | accuracy: 76.29 -> 76.73 | Helped |
Limitations
- Efficiency claims (FLOPs, sparse activation) are asserted without quantitative measurements.
- Gains are reported as relatively small margins over strong baselines and the paper does not report variance across seeds.
- Paper text is internally inconsistent in places (e.g., Table 1 text claims vs. table values, 'learnable' removal), limiting clarity of the method description.
Experimental setup
Datasets: FGVC VTAB-1K Med-VTAB ADE20K Kvasir-seg ISIC skin lesion
Benchmarks: FGVC VTAB-1K Med-VTAB ADE20K Kvasir-seg ISIC
Metrics: accuracy AUC mIoU
Baselines: VPT VPT-Deep GaPT LSPT EXPRES SNF Bi-AdaptFormer
Builds on: Visual Prompt Tuning (VPT) LSPT GaPT EXPRES
Closest papers by meaning
- Federated Disentangled Tuning with Textual Prior Decoupling and Visual Dynamic Adaptation (ICML 2025 · similarity 0.86)
- Minimal Interaction Seperated Tuning: A New Paradigm for Visual Adaptation (CVPR 2025 · similarity 0.86)
- Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptation (NeurIPS 2024 · similarity 0.86)
- Revisiting the Power of Prompt for Visual Tuning (ICML 2024 · similarity 0.86)
- Unleashing the Power of Meta-tuning for Few-shot Generalization Through Sparse Interpolated Experts (ICML 2024 · similarity 0.86)
- Mixture of Nested Experts: Adaptive Processing of Visual Tokens (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.