Research map / Label-efficient and robust learning / OOD and anomaly detection
MetaOOD: Automatic Selection of OOD Detection Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MetaOOD, a zero-shot unsupervised meta-learning framework that selects among 11 OOD detectors by regressing their historical AUROC from language-model embeddings of dataset and method descriptions with an XGBoost meta-predictor.
Problem
Selecting the best OOD detection model for a new image dataset pair without labels or test-time evaluation, which existing outlier-detection meta-learners do not handle well.
Approach
Treat OOD detector selection as a meta-learning regression: an XGBoost meta-predictor maps language-model embeddings of ID/OOD dataset descriptions and detector descriptions to historical AUROC, then picks the top predicted detector for a new dataset pair.
Key results
- MetaOOD attains the lowest average rank among all 11 baselines and 12 methods in total, with Wilcoxon signed-rank p-values below 0.05 against every baseline (e.g., p=0.0357 vs. MetaOOD_0, p<0.0001 vs. ME, MSP, ODIN).
- Language-model embeddings (BERT-based all-mpnet-base-v2) performed slightly better than LLaMA-2-7B and OpenAI text-embedding-3-small variants, and tree-based XGBoost beat a two-layer MLP meta-predictor.
- Openmax was the globally best single detector and GPT-4o mini consistently selected it, ranking in the top 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MetaOOD instead of MetaOOD_0 with handcrafted meta-features and one-hot model embeddings | OOD detector selection · 24 OOD test dataset pairs | selected detector performance rank | Helped |
| Use BERT-based all-mpnet-base-v2 embeddings instead of LLM-generated embeddings (ablation) | MetaOOD · 24 OOD test dataset pairs | selected detector performance rank | Helped |
| Use BERT-based language embeddings instead of handcrafted meta-features and one-hot model embeddings (ablation) | MetaOOD · 24 OOD test dataset pairs | selected detector performance rank | Helped |
| Use XGBoost instead of a two-layer MLP meta-predictor (ablation) | MetaOOD · 24 OOD test dataset pairs | selected detector performance rank | Helped |
| Combine statistical and landmarker meta-features with language embeddings (ablation) | MetaOOD · 24 OOD test dataset pairs | selected detector performance rank | No clear effect |
| Rephrase dataset descriptions while preserving their meaning (ablation) | MetaOOD · 24 OOD test dataset pairs | selected detector performance rank | No clear effect |
Surprising findings
- GPT-4o mini as a zero-shot selector ranked in the top 3, close to MetaOOD, without any meta-learning.
- Mega Ensemble (averaging all detectors' scores) performed poorly, indicating that indiscriminate ensembling hurts.
Limitations
- Depends on the availability and quality of historical dataset pairs; may be less effective when such data are sparse or dissimilar.
- Currently limited to single-modality (image) data and top-1 selection.
- Restricted testbed of 11 detectors and image datasets.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 ImageNet FashionMNIST MNIST Places365 SVHN Textures TIN SSB_hard NINCO iNaturalist
Benchmarks: MetaOOD testbed of 24 ID-OOD test dataset pairs
Metrics: AUROC of selected detector average rank among 12 methods Wilcoxon signed-rank p-value
Baselines: MSP ODIN Mega Ensemble Random selection Global Best ISAC ARGOSMART ALORS NCF MetaOOD_0 GPT-4o mini
Builds on: MetaOD ALORS ISAC ARGOSMART NCF
Closest papers by meaning
- MultiOOD: Scaling Out-of-Distribution Detection for Multiple Modalities (NeurIPS 2024 · similarity 0.90)
- Enhancing the Power of OOD Detection via Sample-Aware Model Selection (CVPR 2024 · similarity 0.87)
- Can We Ignore Labels in Out of Distribution Detection? (ICLR 2025 · similarity 0.86)
- OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable? (ICML 2025 · similarity 0.86)
- Fast Decision Boundary based Out-of-Distribution Detector (ICML 2024 · similarity 0.86)
- DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection (CVPR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.