Research map / Label-efficient and robust learning / Domain adaptation and generalization
Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Self-Feedback Training (SFT), which alternates a feedback phase measuring sharpness inconsistency across training domains with a refinement phase that trains a landscape refiner producing soft labels, to find consistent flat minima for domain generalization.
Problem
Domain generalization fails when flat minima found on training domains do not stay flat/low-loss on unseen test domains, because loss landscapes differ across domains.
Approach
Iteratively refines loss landscapes via a learned landscape refiner that outputs dynamic soft labels (projected via a KL-based label-space projection), guided by a feedback signal measuring the difference in loss sharpness between held-out and trained training domains, so the SAM-found minima become consistent across domains.
Key results
- SFT averages 67.1% over five DomainBed datasets with ResNet-50, versus 64.5% for SAM and 65.9% for MIRO.
- With ViT-B/16 (visual prompt tuning), SFT reaches 77.8% average versus 76.3% for SAM.
- Ablation on OfficeHome raises average accuracy from 84.0% (ERM-style one-hot baseline) to 86.5% with the full landscape-refiner setup.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace SAM with SFT | ResNet-50 · Five DomainBed benchmarks (average) | accuracy: 64.5% → 67.1% | Helped |
| replace SAM with SFT | ViT-B/16 · Five DomainBed benchmarks (average) | accuracy: 76.3 → 77.8 | Helped |
| replace SAM with SFT | ViT-L/14 · Five DomainBed benchmarks (average) | accuracy: 80.3 → 81.2 | Helped |
| add the λ1 sharpness penalty, λ2 consistency penalty, and PCE loss (ablation) | SFT with generated labels on OfficeHome · OfficeHome | accuracy: 85.0 → 86.5 | Helped |
| add the λ2 landscape-consistency penalty (ablation) | SFT with generated labels and λ1=0.5, without PCE · OfficeHome | accuracy: 85.6 → 86.0 | Helped |
| add the λ1 sharpness penalty (ablation) | SFT with generated labels and λ2=0.7, without PCE · OfficeHome | accuracy: 85.8 → 86.0 | Helped |
| add the PCE loss (ablation) | SFT with generated labels, λ1=0.5, and λ2=0.7 · OfficeHome | accuracy: 86.0 → 86.5 | Helped |
Surprising findings
- Label smoothing combined with SAM yields small gains over one-hot labels, while the consistency-guided soft labels give a larger improvement.
Limitations
- SFT currently applies only basic SAM, not advanced SAM variants.
- Large soft-label weight lambda degrades label correctness and accuracy, requiring tuning.
- Theoretical analysis is deferred to the supplementary file.
Experimental setup
Datasets: PACS VLCS OfficeHome TerraIncognita DomainNet
Benchmarks: PACS VLCS OfficeHome TerraIncognita DomainNet
Metrics: out-of-domain accuracy average accuracy
Baselines: ERM SAM GAM GSAM FAD SAGM MIRO SWAD mDSDI IRM CORAL MMD
Builds on: sharpness-aware minimization label smoothing visual prompt tuning DomainBed
Closest papers by meaning
- Unknown Domain Inconsistency Minimization for Domain Generalization (ICLR 2024 · similarity 0.91)
- QT-DoG: Quantization-Aware Training for Domain Generalization (ICML 2025 · similarity 0.90)
- Gradient-Guided Annealing for Domain Generalization (CVPR 2025 · similarity 0.89)
- Sharpness-Aware Model-Agnostic Long-Tailed Domain Generalization (AAAI 2024 · similarity 0.89)
- Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.88)
- Lost Domain Generalization Is a Natural Consequence of Lack of Training Domains (AAAI 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.