Research map / Label-efficient and robust learning / OOD and anomaly detection
OOD-Chameleon: Is Algorithm Selection for OOD Generalization Learnable?
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes OOD-Chameleon, a multi-label MLP classifier that predicts which of five OOD training algorithms (ERM, GroupDRO, oversample, undersample, logits adjustment) is suitable for a new dataset from dataset descriptors alone, trained on a semi-synthetic meta-dataset of resampled datasets.
Problem
Choosing the right OOD-generalization training algorithm for a given dataset normally requires trial-and-error training of many models; the paper asks whether this a priori algorithm selection can be learned from dataset characteristics under label shift, covariate shift, and spurious correlation.
Approach
Frame algorithm selection as multi-label classification over candidate algorithms, trained on a "dataset of datasets" built by resampling fine-grained annotated datasets (CelebA, CivilComments) to controlled shift degrees, so that a selector predicts suitable algorithms from cheap dataset descriptors without training target models.
Key results
- On synthetic tasks, OOD-Chameleon reaches 86.3% 0-1 accuracy and 19.9% WG error vs. 72.5% / 22.7% for Global best and 79.7% / 20.4% for Regression (oracle 100% / 19.0%).
- On CelebA with ResNet18 features, OOD-Chameleon achieves 47.7% WG error vs. 51.3% for Global best and 57.8% for ERM; the selector transfers to MetaShift (39.0% WG error with ResNet18).
- A decision-tree selector performs almost as well as the MLP (73.1% vs. 75.0% accuracy on CelebA ResNet18), and a linear model performs significantly worse (63.4%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use OOD-Chameleon instead of the strongest fixed selector baseline | Synthetic task algorithm selection · Synthetic tasks | WG error (%): 23.9$_{\pm 0.2}$ -> 19.9$_{\pm 0.1}$ | Helped |
| Adaptively select algorithms with OOD-Chameleon rather than always using one fixed algorithm | OOD-Chameleon · CelebA (ResNet18) | WG error: 49.8 $_{\pm 0.5}$ -> 47.7 $_{\pm 0.2}$ | Helped |
| Use an MLP selector instead of a linear selector (ablation) | OOD-Chameleon on CelebA · CelebA (ResNet18) | WG error: 49.6 $_{\pm 0.3}$ -> 47.7 $_{\pm 0.2}$ | Helped |
| Use an MLP selector instead of k-NN (ablation) | OOD-Chameleon on CelebA · CelebA (ResNet18) | WG error: 49.3 $_{\pm 0.2}$ -> 47.7 $_{\pm 0.2}$ | Helped |
| Use an MLP selector instead of a decision tree (ablation) | OOD-Chameleon on CelebA · CelebA (ResNet18) | WG error: 48.1 $_{\pm 0.3}$ -> 47.7 $_{\pm 0.2}$ | Helped |
| Use top prediction logits rather than randomly choosing among binary-positive algorithms (ablation) | OOD-Chameleon at test time · CelebA (ResNet) | WG error: 48.5 -> 47.7 | Helped |
| Vary the algorithm suitability threshold from 0.05 to 0.0 (ablation) | OOD-Chameleon · CelebA (ResNet) | WG error: 47.7 -> 48.3 | Hurt |
| Vary the algorithm suitability threshold from 0.05 to 0.025 (ablation) | OOD-Chameleon · CelebA (ResNet) | WG error: 47.7 -> 48.1 | Hurt |
| Vary the algorithm suitability threshold from 0.05 to 0.10 (ablation) | OOD-Chameleon · CelebA (ResNet) | WG error: 47.7 -> 48.0 | Hurt |
| Train task models by fine-tuning rather than linear probing (ablation) | OOD-Chameleon on CelebA · CelebA tasks | WG error: 47.7 $_{\pm 0.2}$ -> 34.8 $_{\pm 0.4}$ | Helped |
| Select the best algorithm per task rather than random selection | OOD-Chameleon's algorithm selection task · Synthetic tasks | WG error (%): 24.0$_{\pm 0.1}$ -> 19.9$_{\pm 0.1}$ | Helped |
Surprising findings
- A linear selector and a k-NN selector both perform much worse than the MLP, indicating the mapping is non-linear and not just memorization.
- Naive dataset descriptors (trivial properties like number of classes) yield near-random or poor selection, e.g. 52.1% accuracy on synthetic tasks versus 86.3% for the proposed descriptors.
Limitations
- Only five candidate algorithms, all from simple families, and no hyperparameter search.
- Dataset descriptors are hand-designed and depend on attribute annotations; the availability proxy for real data is heuristic.
- Shift taxonomy is limited to binary classes and binary attributes; evaluation is on semi-synthetic rather than real-world WILDS-type tasks.
Experimental setup
Datasets: CelebA CivilComments MetaShift OfficeHome Colored-MNIST MultiNLI synthetic Gaussian data
Benchmarks: synthetic Gaussian tasks CelebA tasks MetaShift OfficeHome Colored-MNIST CivilComments MultiNLI
Metrics: 0-1 accuracy worst-group error averaged-group error
Baselines: Random selection Global best Naive descriptors Regression Oracle selection ERM GroupDRO Oversample Undersample Logits adjustment
Builds on: GroupDRO logits adjustment
Closest papers by meaning
- Unraveling the Key Components of OOD Generalization via Diversification (ICLR 2024 · similarity 0.89)
- Bridging OOD Detection and Generalization: A Graph-Theoretic View (NeurIPS 2024 · similarity 0.88)
- AHA: Human-Assisted Out-of-Distribution Generalization and Detection (NeurIPS 2024 · similarity 0.88)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.88)
- CRoFT: Robust Fine-Tuning with Concurrent Optimization for OOD Generalization and Open-Set OOD Detection (ICML 2024 · similarity 0.87)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.