Research map / Label-efficient and robust learning / Recognition and long-tail
Correlative and Discriminative Label Grouping for Multi-Label Visual Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ML-VPT, a parameter-efficient multi-label visual prompt tuning method that groups classes by co-occurrence and mutual exclusivity, uses group-specific prompt tokens in a ViT, and applies group-aware mixture-of-experts to produce label-aware representations.
Problem
Multi-label image classification models that overemphasize label co-occurrence overfit and perform suboptimally, and full fine-tuning of large pre-trained ViTs is costly.
Approach
Partition classes into co-occurrence (CO) and discriminative (DC) groups via spectral clustering on co-occurrence/dis-occurrence graphs, assign group-specific prompt tokens inside ViT for each group type, and use group-aware MoE gating per class to map group representations to label-aware representations, with dual classifiers balancing CO and DC predictions.
Key results
- On COCO at 224x224 with DINOv2/B, ML-VPT reaches 87.5 mAP versus 86.3 for E2VPT and 86.1 for VPT (ViT-B-21k results also reported).
- Best COCO mAP of 90.6% is achieved with DINOv2/B at 448x448, outperforming VPT's 89.7.
- On VOC07 at 224x224 with ViT-B, ML-VPT gets 92.9 mAP versus 90.4 for VPT.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add CO and DC group prompts (GVPT) to vanilla VPT (ablation) | VPT · COCO | mAP: 78.20 -> 79.61 | Helped |
| Add group-aware MoE to GVPT (ablation) | GVPT · COCO | mAP: 79.61 -> 80.60 | Helped |
| Add CO and DC group prompts (GVPT) to vanilla VPT (ablation) | VPT · VOC07 | mAP: 90.78 -> 92.55 | Helped |
| Add group-aware MoE to GVPT (ablation) | GVPT · VOC07 | mAP: 92.55 -> 93.03 | Helped |
| Use CO grouping only instead of vanilla VPT (ablation) | VPT · COCO | mAP: 77.38 -> 77.79 | Helped |
| Use DC grouping only instead of vanilla VPT (ablation) | VPT · COCO | mAP: 77.38 -> 77.96 | Helped |
| Use both CO and DC grouping instead of vanilla VPT (ablation) | VPT · COCO | mAP: 77.38 -> 78.84 | Helped |
| Use CO grouping only instead of vanilla VPT (ablation) | VPT · VOC07 | mAP: 90.94 -> 91.70 | Helped |
| Use DC grouping only instead of vanilla VPT (ablation) | VPT · VOC07 | mAP: 90.94 -> 91.72 | Helped |
| Use both CO and DC grouping instead of vanilla VPT (ablation) | VPT · VOC07 | mAP: 90.94 -> 92.55 | Helped |
| Vary the number of CO and DC groups (ablation) | ML-VPT · COCO and VOC07 | mAP | Mixed |
| Vary the number of experts per group (ablation) | ML-VPT · COCO and VOC07 | mAP | Mixed |
Surprising findings
- GateVPT and E2VPT, modifications of VPT, do not consistently outperform vanilla VPT on MLC.
- Group-aware MoE gives limited gain on VOC07 (about 0.48 mAP), attributed to few classes per group.
Limitations
- Gains on VOC07 from the group-aware MoE are small because groups contain few classes.
- Performance depends on number of groups and experts, which needs tuning per dataset.
- Compared methods are adapted from single-label VPT variants rather than dedicated MLC PEFT methods.
Experimental setup
Datasets: MS-COCO 2014 Pascal VOC 2007 NUS-WIDE Visual Genome
Benchmarks: MS-COCO 2014 Pascal VOC 2007 NUS-WIDE Visual Genome
Metrics: mAP CP CR CF1 OP OR OF1
Baselines: VPT GateVPT E2VPT Fine Tuning Linear Probing
Builds on: VPT ASL loss BoostMLC label grouping
Closest papers by meaning
- LAMM: Label Alignment for Multi-Modal Prompt Learning (AAAI 2024 · similarity 0.88)
- Multi-Label Few-Shot Image Classification via Pairwise Feature Augmentation and Flexible Prompt Learning (AAAI 2025 · similarity 0.88)
- In Pursuit of Causal Label Correlations for Multi-label Image Recognition (NeurIPS 2024 · similarity 0.87)
- Beyond Words: Augmenting Discriminative Richness via Diffusions in Unsupervised Prompt Learning (CVPR 2025 · similarity 0.86)
- AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024 · similarity 0.86)
- Counterfactual Reasoning for Multi-Label Image Classification via Patching-Based Training (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.