Research map / Label-efficient and robust learning / Domain adaptation and generalization
Weight Diffusion for Future: Learn to Generalize in Non-Stationary Environments
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes W-Diff, a conditional diffusion model over classifier weights that learns the evolution of classifiers across sequentially arriving domains for evolving domain generalization in the domain-incremental setting.
Problem
Evolving domain generalization when source domains arrive sequentially (domain-incremental) and historical data is unavailable, so the model must generalize to future unseen domains.
Approach
Model the evolving pattern at the parameter level: a conditional diffusion model, conditioned on historical classifier weights (reference points) and current class prototypes, generates residual classifier weights toward the current domain's classifier (anchor point). A domain-shared encoder is learned via prediction-consistency loss across stored classifiers, and inference uses an ensemble of gen
Key results
- W-Diff achieves 70.40% OOD worst accuracy on Huffpost and 50.70% on Arxiv, surpassing EvoS (70.19%, 50.19%).
- On RMNIST, W-Diff reaches 87.36% OOD worst accuracy and 94.12% OOD average, versus EvoS 87.04% and 93.84%.
- On 2-Moons, error rate drops to 1.5% from EvoS's 2.5%; on ONP, 32.9% vs EvoS 33.1%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add parameter-level conditional weight diffusion and weight ensemble | domain-incremental evolving domain generalization · Huffpost, Arxiv, RMNIST, fMoW, 2-Moons, ONP | OOD worst accuracy or error rate: 70.19 -> 70.40; 50.19 -> 50.70; 87.04 -> 87.36; 46.24 -> 46.51; 2.5±1.0 -> 1.5±1.0; 33.1±0.6 -> 32.9±0.5 | Helped |
| add parameter-level conditional weight diffusion and weight ensemble | domain-incremental evolving domain generalization · Yearbook OOD worst | accuracy: 94.78 -> 94.05 | Hurt |
| add prediction consistency loss among classifiers (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 91.69 -> 94.12 | Helped |
| add prediction consistency loss among classifiers (ablation) | W-Diff · RMNIST | OOD worst accuracy: 81.87 -> 87.36 | Helped |
| inject diffusion condition through hybrid conditioning rather than cross-attention only (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 91.68 -> 94.12 | Helped |
| inject diffusion condition through hybrid conditioning rather than cross-attention only (ablation) | W-Diff · RMNIST | OOD worst accuracy: 83.19 -> 87.36 | Helped |
| inject diffusion condition through hybrid conditioning rather than input concatenation only (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 92.93 -> 94.12 | Helped |
| inject diffusion condition through hybrid conditioning rather than input concatenation only (ablation) | W-Diff · RMNIST | OOD worst accuracy: 85.03 -> 87.36 | Helped |
| condition generated weights on historical reference points and prototypes rather than using historical weights directly (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 93.70 -> 94.12 | Helped |
| condition generated weights on historical reference points and prototypes rather than adding small uniform noise (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 93.78 -> 94.12 | Helped |
| scale the reference point using timestamp difference (ablation) | W-Diff · RMNIST | OOD avg. accuracy: 94.12 -> 94.17 | No clear effect |
Surprising findings
- W-Diff does not achieve state-of-the-art on Yearbook, where feature-level EvoS performs better, which the authors attribute to feature-level modeling suiting that dataset.
- Injecting the condition only via cross-attention or only via input concatenation performs worse than the hybrid conditioning.
Limitations
- Only classification tasks are considered.
- Only classifier weights are modeled by the diffusion model, not the full network, due to training cost.
- Failures can occur under significant distribution shifts.
Experimental setup
Datasets: Huffpost Arxiv Yearbook RMNIST fMoW 2-Moons ONP
Benchmarks: Huffpost Arxiv Yearbook RMNIST fMoW 2-Moons ONP
Metrics: accuracy error rate OOD average accuracy OOD worst accuracy
Baselines: IRM CORAL Mixup LISA GI EWC SI A-GEM DRAIN EvoS LSSAE CDOT
Builds on: latent diffusion model p-diff EvoS DRAIN
Closest papers by meaning
- Enhancing Evolving Domain Generalization through Dynamic Latent Representations (AAAI 2024 · similarity 0.89)
- Self-Training with Dynamic Weighting for Robust Gradual Domain Adaptation (NeurIPS 2025 · similarity 0.88)
- Gradient-Guided Annealing for Domain Generalization (CVPR 2025 · similarity 0.87)
- Transfer Learning for Diffusion Models (NeurIPS 2024 · similarity 0.87)
- Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model (ICLR 2025 · similarity 0.87)
- Diffusion-based Neural Network Weights Generation (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.