Research map / Label-efficient and robust learning / OOD and anomaly detection
Towards Robust Out-of-Distribution Generalization Bounds via Sharpness
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Derives a robustness-based OOD generalization bound and links algorithmic robustness to loss-landscape sharpness for random-feature ReLU networks, supported by ridge regression and vision experiments.
Problem
Existing OOD generalization bounds (VC, Rademacher, PAC-Bayes, domain divergence) ignore the optimization properties of the learned model, such as sharpness of the minimum, and are loose for overparameterized networks.
Approach
Replaces divergence-based OOD terms with a partition-based algorithmic robustness measure, bounds the robustness constant by the Hessian-trace sharpness of the learned minimum (for random-feature ReLU networks), yielding a sharpness-based OOD bound that does not depend on hypothesis size.
Key results
- On RotatedMNIST, the paper reports a clear trend that flatter minima (lower sharpness) yield better OOD accuracy across 4 held-out rotation environments, with 6 seeds per algorithm.
- In ridge regression, larger L2 penalty beta lowers both sharpness and the OOD test-loss increase under angular distribution shift.
- On the synthetic spurious-feature task, the proposed robust bound stays flat as model size grows past the overparameterized regime, while the baseline domain-shift bound keeps increasing; the paper reports the proposed bounds are tighter than the baseline.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| increase ridge-regression L2 regularization β | Ridge regression · Distributional shift as α increases | OOD test loss | Helped |
| increase ridge-regression L2 regularization β | Ridge regression · Training distribution α=0 | sharpness | Hurt |
| use a sharper minimum | ERM, SWAD, Mixup, and GroupDRO models · RotatedMNIST OOD environments | OOD accuracy | Hurt |
| increase model size | Random-feature logistic regression · Synthetic spurious-feature experiment, p_maj = 0.9 | worst-group error | No clear effect |
| use the proposed robustness-based bound instead of the baseline domain-shift bound | OOD generalization bound · Synthetic spurious-feature experiment across model sizes | bound concentration error | Helped |
| use the proposed distribution-distance and generalization bounds instead of the baseline | OOD generalization bound · Synthetic spurious-feature experiment across distribution shifts | bound tightness | Helped |
| increase model sharpness | ERM models · Wilds-Camelyon17 OOD test environment | OOD accuracy | Hurt |
| increase model sharpness | ERM models · PACS OOD test environments | OOD accuracy | Hurt |
Surprising findings
- Sharper minima hurt OOD generalization in the reported experiments, which is contrary to the common view that sharpness is only an ID-generalization concern.
Limitations
- The sharpness-robustness connection is proven only for one-hidden-layer random-feature ReLU networks with strong assumptions (Hessian Lipschitz per partition, rotationally invariant features).
- Robustness constant epsilon is intractable to compute, and the partition count K must be chosen; the bound is only as tight as this proxy.
- The paper notes the dependence between robustness and sharpness is not exact and a unified theorem for general function classes is still needed.
Experimental setup
Datasets: RotatedMNIST PACS Wilds-Camelyon17 synthetic ridge regression data synthetic spurious-feature data
Benchmarks: RotatedMNIST PACS Wilds-Camelyon17 synthetic spurious-feature task
Metrics: OOD test accuracy test loss worst-group error sharpness generalization bound value
Baselines: ERM SWAD Mixup GroupDRO Zhao et al. domain divergence bound
Builds on: algorithmic robustness (Xu & Mannor) Zhao et al. domain adaptation bound DomainBed
Closest papers by meaning
- Generalization Bounds for Out-of-distribution Generalization (ICML 2026 · similarity 0.92)
- Models Out of Line: A Fourier Lens on Distribution Shift Robustness (NeurIPS 2022 · similarity 0.90)
- A Modern Look at the Relationship between Sharpness and Generalization (ICML 2023 · similarity 0.90)
- Robustness Implies Generalization via Data-Dependent Generalization Bounds (ICML 2022 · similarity 0.90)
- In Search of Forgotten Domain Generalization (ICLR 2025 · similarity 0.88)
- The Best of Both Worlds: On the Dilemma of Out-of-distribution Detection (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.