Research map / Label-efficient and robust learning / Domain adaptation and generalization
PEER Pressure: Model-to-Model Regularization for Single Source Domain Generalization
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PEER, a model-to-model regularization for augmentation-based single source domain generalization that averages parameters of a task model with a proxy model trained under mutual-information-based feature regularization, reducing mid-training OOD accuracy fluctuation.
Problem
Augmentation-based single source domain generalization shows large fluctuations in target-domain accuracy during training, making model selection unreliable, because augmentation distorts learned features and the model fails to accumulate knowledge across augmentations.
Approach
A frozen task model regularizes a trainable proxy model by maximizing mutual information between their intermediate representations (via InfoNCE or Barlow Twins-style losses), and the task model is periodically updated by parameter-averaging over the proxy's trajectory, so it accumulates knowledge across diverse augmentations without an external pre-trained teacher.
Key results
- PEER with random augmentation reaches 81.06% average accuracy on Digits versus 73.98% for RandAug and 76.89% for SimDE (reported).
- On Office-Home PEER reaches 60.63% average accuracy, a 10.62-point gain over RandAug, and on VLCS 79.10% (6.66-point gain over RandAug).
- PEER lowers mid-train OOD fluctuation across all benchmarks, e.g. PACS average variance from 4.02 (RandAug) to 3.59.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add PEER regularization and parameter averaging to random augmentation | RandAug · PACS target domains | average accuracy: 55.59 -> 59.42 | Helped |
| add PEER regularization and parameter averaging to random augmentation | RandAug · Digits target domains | average accuracy: 73.98 -> 81.06 | Helped |
| add PEER regularization and parameter averaging to random augmentation | RandAug · Office-Home target domains | average accuracy: 50.01 -> 60.63 | Helped |
| add PEER regularization and parameter averaging to random augmentation | RandAug · VLCS target domains | average accuracy: 72.44 -> 79.10 | Helped |
| add parameter averaging to PEER (ablation) | PEER · PACS target domains | average accuracy: 55.25 -> 59.42 | Helped |
| add parameter averaging to PEER (ablation) | PEER · Digits target domains | average accuracy: 74.46 -> 81.06 | Helped |
| add PEER regularization to parameter-space ensembling (ablation) | parameter-space ensemble · PACS target domains | average accuracy: 53.51 -> 59.42 | Helped |
| add PEER regularization to parameter-space ensembling (ablation) | parameter-space ensemble · Digits target domains | average accuracy: 80.72 -> 81.06 | Helped |
Surprising findings
- Parameter-space ensembling without the PEER regularizer gives only marginal generalization gains on PACS (53.51 vs 59.42 with PEER), indicating averaging alone is insufficient.
- Higher-complexity augmentations improve target accuracy but also increase mid-train OOD fluctuation, so accuracy and stability trade off.
Limitations
- The paper's gains on standard benchmarks are reported against backbones (AlexNet) that differ from some compared methods (SimDE, AdvST use ResNet-18), complicating direct comparison.
- The method's regularization hyperparameters (w, lambda, k) are tuned per setup and selected via ablations.
Experimental setup
Datasets: PACS Digits (MNIST, SVHN, MNIST-M, SYNDIGIT, USPS) Office-Home VLCS
Benchmarks: PACS Digits Office-Home VLCS
Metrics: target domain accuracy variance of target domain accuracy (OOD fluctuation) source-target OTDD distance CKA similarity
Baselines: ERM ADA M-ADA L2D PDEN SimDE AdvST MetaCNN RandAug
Builds on: RandAug InfoNCE Barlow Twins Model soups / weight averaging PDEN
Closest papers by meaning
- In Search of Lost Domain Generalization (ICLR 2021 · similarity 0.87)
- AdvST: Revisiting Data Augmentations for Single Domain Generalization (AAAI 2024 · similarity 0.86)
- LFME: A Simple Framework for Learning from Multiple Experts in Domain Generalization (NeurIPS 2024 · similarity 0.86)
- Unknown Domain Inconsistency Minimization for Domain Generalization (ICLR 2024 · similarity 0.86)
- Enhancing Evolving Domain Generalization through Dynamic Latent Representations (AAAI 2024 · similarity 0.86)
- Characterizing the Generalization Error of Random Feature Regression with Arbitrary Data-Augmentation (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.