Research map / Label-efficient and robust learning / Test-time adaptation
Domain-wise Data Acquisition to Improve Performance under Distribution Shift
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Domain-wise Active Acquisition (DAA), which iteratively reweights how a fixed sample budget is allocated across training domains, using MMD between accumulated samples and unlabeled test data, to improve generalization under covariate shift.
Problem
Under covariate distribution shift, how to allocate a limited labeled-data acquisition budget across heterogeneous training domains so the trained model generalizes to an unlabeled target test domain.
Approach
Instead of sample-level selection from a pool, the method controls acquisition at the domain level and splits the budget into K rounds; each round the domain weights are learned by minimizing the embedding distance (MMD-style) between the weighted mixture of acquired training data and the unlabeled test mean embedding, and the next batch is drawn proportionally to these weights.
Key results
- On Rotated MNIST with 600 samples, DAA outperforms uniform acquisition and matches or exceeds the best hand-picked domain combination [60, 120] for ERM, reweighting and self-supervised models.
- On FMOW with 6000 samples, DAA reaches 87.51% (ERM), 88.25% (reweighting) and 88.11% (self-supervised) versus 86.51%, 87.66% and 87.47% for uniform acquisition.
- On ACS PUMS, DAA achieves the best accuracy in almost all task/state cases across ERM, reweighting and self-supervised learning.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace uniform acquisition with DAA | ResNet-50 training on FMOW · FMOW, 2000 samples | accuracy: 83.08% -> 84.85%; 84.49% -> 85.10%; 83.96% -> 85.28% | Helped |
| Replace uniform acquisition with DAA | ResNet-50 training on FMOW · FMOW, 4000 samples | accuracy: 85.23% -> 86.84%; 84.56% -> 86.25%; 86.28% -> 87.35% | Helped |
| Replace uniform acquisition with DAA | ResNet-50 training on FMOW · FMOW, 6000 samples | accuracy: 86.51% -> 87.51%; 87.66% -> 88.25%; 87.47% -> 88.11% | Helped |
| Use more acquisition rounds (ablation) | DAA on ACS PUMS · ACS PUMS, Income and PublicCoverage tasks | accuracy | Mixed |
| Replace MSE distance with MMD distance for acquisition (ablation) | DAA on ACS PUMS · ACS PUMS, Income and PublicCoverage tasks | accuracy | Helped |
Surprising findings
- Model-level advances (reweighting, self-supervised learning) cannot fully compensate for poor training-data acquisition; ERM with a good domain combination can beat self-supervised learning with a weaker combination.
Limitations
- Assumes covariate shift with a shared conditional label distribution P(Y|X).
- Relies on a finite number of acquisition rounds; too few or too many rounds hurt performance, and sampling noise in weight estimates is not fully resolved.
- Evaluated on only three datasets with a small set of training domains.
Experimental setup
Datasets: Rotated MNIST ACS PUMS FMOW
Benchmarks: Rotated MNIST ACS PUMS FMOW
Metrics: prediction accuracy MMD distance
Baselines: Uniform acquisition Top-K domain selection ERM Sample reweighting Self-supervised learning
Builds on: sample reweighting Empirical Risk Minimization self-supervised pseudo-labeling
Closest papers by meaning
- Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains (AAAI 2024 · similarity 0.86)
- Rethinking the Evaluation Protocol of Domain Generalization (CVPR 2024 · similarity 0.85)
- Learning Neural Networks with Distribution Shift: Efficiently Certifiable Guarantees (ICLR 2025 · similarity 0.85)
- Tolerant Algorithms for Learning with Arbitrary Covariate Shift (NeurIPS 2024 · similarity 0.85)
- Improving Domain Generalization with Domain Relations (ICLR 2024 · similarity 0.85)
- COME: Test-time Adaption by Conservatively Minimizing Entropy (ICLR 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.