Research map / Label-efficient and robust learning / Recognition and long-tail
OpenworldAUC: Towards Unified Evaluation and Optimization for Open-world Prompt Tuning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OpenworldAUC, a pairwise ranking metric unifying base-to-new detection and classification for open-world prompt tuning, plus Gated Mixture-of-Prompts (GMoP) to optimize it on CLIP.
Problem
Evaluating open-world prompt tuning of vision-language models requires jointly assessing base/new domain detection and classification, insensitive to unknown base/new sample ratios; existing metrics (HM, OverallAcc, AUROC) fail to satisfy all these properties.
Approach
A pairwise joint-probability metric (OpenworldAUC) that counts a base-new pair as correct only if the detector ranks the base sample above the new one and both are correctly classified; optimized by Gated Mixture-of-Prompts with domain-specific prompts and a sigmoid gate that routes an AUC-style ranking loss over correctly classified samples, using pseudo base-to-new partitions.
Key results
- GMoP achieves average OpenworldAUC of 60.94 on the 11 open-world recognition benchmarks, versus 58.77 for DeCoOp and 59.00 for DePT-Kg.
- On the 10 domain-imbalanced settings (1:5 ratio in both directions), GMoP averages 68.84 versus 66.87 for DeCoOp.
- On ImageNet source to ImageNet-V2/S/R/A targets, GMoP averages higher OpenworldAUC than all competitors (e.g., 52.64 source, 43.98 on V2).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GMoP instead of the strongest listed baseline | Open-world recognition · 11-benchmark average | OpenworldAUC: 58.90 -> 60.94 | Helped |
| Use GMoP instead of DeCoOp | Open-world recognition · OxfordPets | OpenworldAUC: 88.72 -> 89.77 | Helped |
| Use GMoP instead of the strongest listed baseline | Open-world recognition · Aircraft (AC) | OpenworldAUC: 12.71 -> 11.42 | Hurt |
| Use GMoP instead of the strongest listed baseline | Open-world recognition · EuroSAT (ES) | OpenworldAUC: 56.55 -> 53.09 | Hurt |
| Use GMoP instead of the strongest listed baseline | Open-world recognition · UCF101 (UCF) | OpenworldAUC: 63.39 -> 62.39 | Hurt |
| Use GMoP instead of DeCoOp | Open-world domain generalization · ImageNet-V2 | OpenworldAUC: 43.01 -> 43.98 | Helped |
| Use GMoP instead of DeCoOp | Open-world domain generalization · ImageNet-R | OpenworldAUC: 61.01 -> 62.67 | Helped |
| Increase the number of pseudo base-to-new partitions K (ablation) | GMoP · Flowers102 and SUN397 | OpenworldAUC | Helped |
| Increase the training-shot count (ablation) | GMoP · DTD and OxfordPets | OpenworldAUC | Helped |
| Add cross-entropy regularization to the ranking objective (ablation) | GMoP · Flowers102 and Caltech101 | OpenworldAUC | Helped |
| Use a mixture of specialized prompts instead of a single prompt (ablation) | GMoP · Flowers102, DTD, Caltech101 and ImageNet | OpenworldAUC | Helped |
| Use the adaptive sigmoid gate instead of a fixed 0-1 gate (ablation) | GMoP · Open-world prompt tuning | OpenworldAUC | Helped |
Surprising findings
- OverallAcc increases substantially as the number of base-domain samples grows, showing it is dominated by the larger domain.
- Optimizing the AUC ranking loss from scratch can harm feature representations, so a cross-entropy regularizer with lambda in [1/2, 1] is used.
Limitations
- New-domain classifier is a fixed zero-shot CLIP prompt rather than learned, which limits adaptation.
- Pseudo base-to-new partitions may differ from the real new-domain distribution, a shift captured as a stochastic error term in the bound.
Experimental setup
Datasets: ImageNet Caltech101 OxfordPets Cars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet-V2
Benchmarks: Caltech101 OxfordPets Cars Flowers102 Food101 FGVCAircraft SUN397 DTD EuroSAT UCF101 ImageNet ImageNet-V2
Metrics: OpenworldAUC HM OverallAcc AUROC
Baselines: CLIP CoOp MaPLe PromptSRC LoCoOp KgCoOp DePT Gallop DeCoOp TCP
Builds on: CoOp DeCoOp LoCoOp AUROC
Closest papers by meaning
- DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection (ICML 2024 · similarity 0.90)
- DPC: Dual-Prompt Collaboration for Tuning Vision-Language Models (CVPR 2025 · similarity 0.89)
- Learning to Learn Better Visual Prompts (AAAI 2024 · similarity 0.89)
- Black-Box Test-Time Prompt Tuning for Vision-Language Models (AAAI 2025 · similarity 0.89)
- ZIP: An Efficient Zeroth-order Prompt Tuning for Black-box Vision-Language Models (ICLR 2025 · similarity 0.89)
- IPO: Interpretable Prompt Optimization for Vision-Language Models (NeurIPS 2024 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.