Research map / Label-efficient and robust learning / OOD and anomaly detection
Spurious Feature Diversification Improves Out-of-distribution Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explains why weight-space ensembles like WiSE-FT generalize OOD via diversification of spurious features (FalseFalseTrue phenomenon), proves this in a multi-class bilinear model, and proposes BANG, which balances confidence by using label smoothing/Mixup fine-tuning before averaging with the pre-trained CLIP.
Problem
Why ensemble methods, especially weight space ensembles such as WiSE-FT, improve out-of-distribution generalization is not well understood, and WiSE-FT suffers from over-confidence of the fine-tuned model under distribution shift.
Approach
Ensembles succeed OOD by drawing on a more diverse set of spurious features, so that each spurious feature's individual contribution is weakened and independent mistakes are unlikely to coincide; the weight-space ensemble further amplifies overlapped features differently than output-space ensembles. Relatedly, mismatched confidence between averaged models biases the ensemble, so BANG fine-tunes wi
Key results
- On MultiColorMNIST at p=0.80, the two-model output ensemble reached 55.25% OOD accuracy versus about 48.6-49.3% for individual models.
- BANG (Mixup + LS) achieved 64.9% average OOD accuracy across five ImageNet variants versus 63.0% for WiSE-FT, about 1.9pp improvement.
- SingleColorMNIST ensembles gave essentially no OOD gain over single models, consistent with the theory that diversity of spurious features drives the benefit.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use BANG with Mixup and Label Smoothing instead of vanilla WiSE-FT | CLIP ViT-B/16 fine-tuned on ImageNet · Avg OOD (IN-V2, IN-R, IN-A, IN-S, ObjectNet) | accuracy: 63.0 -> 64.9 | Helped |
| use BANG with Mixup and Label Smoothing instead of vanilla WiSE-FT | CLIP ViT-B/16 fine-tuned on ImageNet · IN-A | accuracy: 52.2 -> 58.2 | Helped |
| add Label Smoothing during fine-tuning before weight averaging (ablation) | WiSE-FT / CLIP ViT-B/16 · Avg OOD (IN-V2, IN-R, IN-A, IN-S, ObjectNet) | accuracy: 63.0 -> 63.9 | Helped |
| add Mixup during fine-tuning before weight averaging (ablation) | WiSE-FT / CLIP ViT-B/16 · Avg OOD (IN-V2, IN-R, IN-A, IN-S, ObjectNet) | accuracy: 63.0 -> 64.7 | Helped |
| ensemble two independently initialized models | 2-layer MLP · MultiColorMNIST, p=0.80 | OOD accuracy: 48.57±0.92 -> 55.25±0.75; 49.26±0.83 -> 55.25±0.75 | Helped |
| ensemble two independently initialized models | 2-layer MLP · MultiColorMNIST, p=0.90 | OOD accuracy: 26.01±0.45 -> 29.28±0.40; 26.63±0.42 -> 29.28±0.40 | Helped |
| ensemble two independently initialized models (ablation) | 2-layer MLP · SingleColorMNIST, p=0.80 | OOD accuracy: 27.90 ± 0.14 -> 27.87 ± 0.22 | No clear effect |
| increase ensemble size from 1 to 16 models (ablation) | 2-layer MLP · MultiColorMNIST, p=0.80 | OOD accuracy: 49.33±2.02 -> 64.85±1.09 | Helped |
| increase ensemble size from 1 to 16 models (ablation) | 2-layer MLP · SingleColorMNIST, p=0.80 | OOD accuracy: 27.90 ± 0.14 -> 27.87 ± 0.22 | No clear effect |
Surprising findings
- WiSE-FT corrects many samples where both pre-trained and fine-tuned models are wrong (the FalseFalseTrue phenomenon), and on IN-R and IN-S the TF+FT improvement contribution is negative.
- Scaling the fine-tuned model's confidence (imbalanced scaling) can cause a substantial OOD performance drop, and temperature scaling tuned on ID data fails to calibrate OOD.
Limitations
- Theory uses infinite samples and a bilinear model with orthogonal features and a small-noise assumption, which may not hold for deep networks.
- WiSE-FT on MultiColorMNIST was left to future work because no suitable pre-trained model exists for the synthetic dataset.
- BANG is evaluated only with CLIP ViT-B/16 on ImageNet and its variants.
Experimental setup
Datasets: ImageNet MultiColorMNIST SingleColorMNIST
Benchmarks: ImageNet ImageNet-V2 ImageNet-R ImageNet-A ImageNet-Sketch ObjectNet MultiColorMNIST SingleColorMNIST
Metrics: accuracy OOD accuracy FalseFalseTrue ratio ImproveContri
Baselines: WiSE-FT zero-shot CLIP fine-tuning output space ensemble temperature scaling
Builds on: WiSE-FT Invariant Risk Minimization linear mode connectivity
Closest papers by meaning
- Aggregation Hides Out-of-Distribution Generalization Failures from Spurious Correlations (NeurIPS 2025 · similarity 0.88)
- Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance (NeurIPS 2024 · similarity 0.87)
- Out-of-Distribution Detection via Deep Multi-Comprehension Ensemble (ICML 2024 · similarity 0.87)
- Unraveling the Key Components of OOD Generalization via Diversification (ICLR 2024 · similarity 0.86)
- Theoretical Limitations of Ensembles in the Age of Overparameterization (ICML 2025 · similarity 0.86)
- Expecting The Unexpected: Towards Broad Out-Of-Distribution Detection (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.