Research map / Learning theory and algorithms / Kernels and regression theory
Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves a minimax lower bound showing poly(1/epsilon) training domains are needed for domain generalization, and provides ColoredMNIST/RotatedMNIST experiments showing test accuracy rises with the number of training domains.
Problem
Explaining why state-of-the-art domain generalization algorithms perform on par with ERM and random guessing on DomainBed, and how many training domains are needed to achieve small excess error on an unseen test domain.
Approach
Attributes lost domain generalization to too few training domains: a two-stage hard-instance construction gives a minimax lower bound on the number of training domains (at least poly(V/epsilon), with Omega(1/epsilon^2) in the n=1 case) for any learner, independent of per-domain sample size.
Key results
- On ColoredMNIST with the oracle selection method, ERM test accuracy rises from about 0.67 at 4 domains to 0.744 at 192 domains.
- Test accuracy of most algorithms is monotonically increasing with the number of training domains, with exceptions for IRM, MMD, and some others, across training-domain validation, oracle, and leave-one-domain-out selection.
- Theorem 1 gives a lower bound RE,1(h,F) >= c min((m-1)/(Eh), sqrt((m-1)/E)), so poly(1/epsilon) training domains are needed for small excess error.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Increase the number of training domains from 4 to 192 | ERM · ColoredMNIST | accuracy: 0.6697±0.0120 -> 0.7437±0.0014 | Helped |
| Increase the number of training domains from 4 to 192 | IRM · ColoredMNIST | accuracy: 0.5500±0.0091 -> 0.7287±0.0012 | Helped |
| Increase the number of training domains from 4 to 192 | GroupDRO · ColoredMNIST | accuracy: 0.6710±0.0134 -> 0.7443±0.0014 | Helped |
| Increase the number of training domains from 4 to 192 | Mixup · ColoredMNIST | accuracy: 0.6081±0.0144 -> 0.7424±0.0013 | Helped |
| Increase the number of training domains from 4 to 192 | MLDG · ColoredMNIST | accuracy: 0.6744±0.0046 -> 0.7461±0.0010 | Helped |
| Increase the number of training domains from 4 to 192 | CORAL · ColoredMNIST | accuracy: 0.6586±0.0139 -> 0.7437±0.0014 | Helped |
| Increase the number of training domains | Domain generalization algorithms, including ERM and IRM · RotatedMNIST | test accuracy | Helped |
| Use MUNIT or VGG11 architecture instead of the original architecture (ablation) | ERM and IRM · ColoredMNIST | test accuracy | Helped |
| Vary the number of samples per domain n from 1000 to 20000 (ablation) | ERM and IRM · ColoredMNIST | test accuracy | Mixed |
Surprising findings
- IRM's accuracy on RotatedMNIST drops slightly beyond about 100 domains, which the authors attribute to limited per-domain samples or randomness.
- A large unexplained dip for ERM on ColoredMNIST near E=100 with VGG11-style runs.
Limitations
- Lower bound assumes n=1 sample per domain and a shared instance support; finite-n case is only bounded via the n=1 result.
- Experiments use only two synthetic MNIST-variant datasets with 5 seeds (1 seed for VGG11) and default hyperparameters, so they only loosely support the theory.
- Theorem's assumption E >= V is not met in some experiments (E < 30), which the authors only discuss empirically.
Experimental setup
Datasets: ColoredMNIST RotatedMNIST
Benchmarks: ColoredMNIST RotatedMNIST
Metrics: test accuracy
Baselines: IRM GroupDRO Mixup MLDG CORAL MMD DANN C-DANN
Builds on: Massart and Nédélec risk bounds Invariant Risk Minimization DomainBed
Closest papers by meaning
- Set Valued Predictions For Robust Domain Generalization (ICML 2025 · similarity 0.89)
- How Many Domains Suffice for Domain Generalization? A Tight Characterization via the Domain Shattering Dimension (NeurIPS 2025 · similarity 0.89)
- Understanding Domain Generalization: A Noise Robustness Perspective (ICLR 2024 · similarity 0.88)
- Sharpness-Aware Model-Agnostic Long-Tailed Domain Generalization (AAAI 2024 · similarity 0.88)
- Rethinking the Evaluation Protocol of Domain Generalization (CVPR 2024 · similarity 0.88)
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.