Research map / Label-efficient and robust learning / Recognition and long-tail
CIFAR-10-Warehouse: Broad and More Realistic Testbeds in Model Generalization Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces CIFAR-10-Warehouse, 180 real-world and diffusion-generated CIFAR-10-class test domains, and benchmarks accuracy prediction and domain generalization methods on it.
Problem
Existing testbeds for out-of-distribution generalization either have few real-world domains or are synthetic image corruptions, which limits evaluation of accuracy prediction (AccP) and domain generalization (DG) methods.
Approach
Build a large collection of 180 CIFAR-10-class domains by prompting seven image search engines (with color, cartoon, and other prompt variations) and Stable Diffusion, yielding broad real-world distribution shifts for AccP and DG benchmarking.
Key results
- MS-AoL achieves the lowest average MAE on CIFAR-10-W across 40 classifiers (4.45% overall), while prediction-score methods and BoP perform best on CIFAR-10-Cs, showing ranking differences.
- Average MAE of AccP methods on ResNet44 is 3.62% on CIFAR-10-Cs versus 6.65% on all CIFAR-10-W sets.
- Domain generalization methods improve over ERM on near-OOD sets but gains shrink on far-OOD cartoon sets; SD with 4 sources reaches 89.01% average accuracy on CIFAR-10-W.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Evaluate accuracy-prediction methods on CIFAR-10-W instead of CIFAR-10 corruption series | Accuracy prediction methods · CIFAR-10-W versus CIFAR-10-Cs (ResNet44) | MAE (%): 3.62% on CIFAR-10-Cs; 6.65% on CIFAR-10-W | Hurt |
| Use MS-AoL rather than the other evaluated accuracy-prediction methods | Accuracy prediction methods · CIFAR-10-W all 180 sets (ResNet44) | MAE (%): 7.96 -> 4.56 | Helped |
| Use cartoon-plus-color test domains (KWC) rather than keyword-plus-color domains (KW) | Accuracy prediction methods · CIFAR-10-W (ResNet44) | MAE (%): 5.26% -> 9.14% | Hurt |
| Train the classifier on CIFAR-10-F rather than CIFAR-10 | Accuracy prediction methods · CIFAR-10-W | MAE | Helped |
| Remove confusing classes such as deer and horse from test sets (ablation) | Prediction-score accuracy-prediction methods · CIFAR-10-W with missing test classes | MAE | Helped |
| Remove single classes such as cat, dog, frog, or ship from test sets (ablation) | Prediction-score accuracy-prediction methods · CIFAR-10-W with missing test classes | MAE | No clear effect |
| Remove test classes (ablation) | Nuclear norm accuracy-prediction method · CIFAR-10-W with missing test classes | MAE | Hurt |
| Reduce the number of test images (ablation) | Accuracy prediction methods · CIFAR-10-W and CIFAR-10-Cs | MAE | Hurt |
| Use four source domains rather than one source domain (ablation) | Domain generalization methods · CIFAR-10-W (ResNet18, SD) | Top-1 accuracy (%): 72.70 ± 4.28 -> 89.01 ± 0.49 | Helped |
| Apply domain generalization methods rather than ERM | Domain generalization methods · Near-OOD CIFAR-10-W domains | Top-1 accuracy | Helped |
| Apply domain generalization methods rather than ERM | Domain generalization methods · Far-OOD CIFAR-10-W domains | Top-1 accuracy | Mixed |
| Use SagNet rather than ERM with three source domains | ResNet18 · CIFAR-10-W KWC domains | Top-1 accuracy (%): approx. +0.56% | Helped |
Surprising findings
- Pred. s (0.9), best on CIFAR-10-Cs for ResNet44, does not remain best on CIFAR-10-W.
- Existing DG methods yield little consistent improvement over ERM on single-source settings, with gains mainly on real search-engine domains.
Limitations
- CIFAR-10 is small and low-resolution (32x32 main setting), and each domain has only 300 to 8,000 images.
- Domain coverage is broad but not exhaustive; only 10 classes.
- MS-AoL incurs significant computational burden from many pairwise model evaluations.
Experimental setup
Datasets: CIFAR-10 CIFAR-10-W CIFAR-10-C CIFAR-10.1 CIFAR-10.2 CIFAR-10-F CINIC
Benchmarks: CIFAR-10-W CIFAR-10-Cs
Metrics: MAE (%) Spearman rank correlation top-1 accuracy (%)
Baselines: Prediction score Entropy ATC-MC DoC Nuclear norm FD BoP ERM SagNet SelfReg SD Fishr
Builds on: Agreement on the Line Spectral Decoupling Frechet distance
Closest papers by meaning
- In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.86)
- Rethinking the Evaluation Protocol of Domain Generalization (CVPR 2024 · similarity 0.86)
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.85)
- Gradient-Guided Annealing for Domain Generalization (CVPR 2025 · similarity 0.84)
- PracticalDG: Perturbation Distillation on Vision-Language Models for Hybrid Domain Generalization (CVPR 2024 · similarity 0.84)
- 4DBInfer: A 4D Benchmarking Toolbox for Graph-Centric Predictive Modeling on RDBs (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.