Research map / Label-efficient and robust learning / Domain adaptation and generalization
Domain-Aware Fine-Tuning: Enhancing Neural Network Adaptability
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Domain-Aware Fine-Tuning (DAFT), which converts batch-norm statistics/parameters to the target domain before fine-tuning and integrates linear probing and fine-tuning into one stage with separate learning rates, to reduce feature distortion.
Problem
Fine-tuning pre-trained networks distorts strong pre-trained features, hurting OOD generalization; LP-FT mitigates this but freezes batch norm in test mode and optimizes the head before the backbone adapts, limiting performance.
Approach
Convert existing BN statistics and affine parameters to target-domain values without adding new ones, so fine-tuning needs only small BN adjustments; and optimize head and feature extractor jointly in a single stage with separately tuned learning rates and zero head initialization.
Key results
- On CIFAR-10 with MoCo, DAFT reaches 94.18% on CIFAR-10.1 and 92.36% on STL versus 93.60% and 90.79% for LP-FT.
- On Cityscapes segmentation, DAFT achieves 76.03 mIoU versus 74.42 for FT and 66.41 for LP-FT.
- Averaged over five tasks, DAFT gives the best ID and OOD accuracy for all three pre-trained backbones (e.g., MoCo OOD 75.66 vs 73.06 for LP-FT).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use DAFT instead of LP-FT | MoCo-TP · fMoW OOD | accuracy (%): 38.50 -> 43.29 | Helped |
| use DAFT instead of LP-FT | MoCo · DomainNet OOD | accuracy (%): 70.89 -> 74.67 | Helped |
| use DAFT instead of LP-FT | MoCo · STL | accuracy (%): 90.79 -> 92.36 | Helped |
| use DAFT instead of FT | MoCo ResNet-50 · VOC | mean IoU (%): 78.47 -> 79.23 | Helped |
| use DAFT instead of FT | MoCo ResNet-50 · Cityscapes | mean IoU (%): 74.42 -> 76.03 | Helped |
| use DAFT instead of LP-FT | MoCo · CIFAR-10-C | mCE (%): 20.0 -> 19.4 | Helped |
| add BN conversion to FT (ablation) | MoCo · CIFAR-10 ID | accuracy (%): 97.66 -> 97.59 | Hurt |
| add BN conversion to FT (ablation) | MoCo · CIFAR-10.1 | accuracy (%): 92.95 -> 93.27 | Helped |
| add BN conversion to FT (ablation) | MoCo · STL | accuracy (%): 81.18 -> 84.05 | Helped |
| add BN conversion to LP-FT (ablation) | MoCo · CIFAR-10 ID | accuracy (%): 97.66 -> 97.31 | Hurt |
| add BN conversion to LP-FT (ablation) | MoCo · CIFAR-10.1 | accuracy (%): 93.60 -> 93.00 | Hurt |
| add BN conversion to LP-FT (ablation) | MoCo · STL | accuracy (%): 90.79 -> 91.66 | Helped |
Surprising findings
- Integrated LP-FT without BN conversion performs similarly to LP-FT, suggesting the single-stage integration alone gives little benefit and BN conversion is key.
Limitations
- Gains on some OOD metrics are small or absent (e.g., CLIP LP is best on Living-17 OOD and FT-based methods vary across datasets).
- Benefits depend on the network containing batch normalization layers; the method is not described for BN-free architectures.
- Experiments use ResNet-50 backbones only, with no transformer models reported.
Experimental setup
Datasets: CIFAR-10 CIFAR-10.1 STL Entity-30 Living-17 DomainNet fMoW Pascal VOC2012 Cityscapes CIFAR-10-C
Benchmarks: CIFAR-10 CIFAR-10.1 STL Entity-30 Living-17 DomainNet fMoW Pascal VOC2012 Cityscapes CIFAR-10-C
Metrics: ID accuracy OOD accuracy mIoU mean Corruption Error cosine similarity L2 distance
Baselines: LP FT LP-FT
Builds on: LP-FT Batch Normalization Side-tuning FLYP
Closest papers by meaning
- Reducing Divergence in Batch Normalization for Domain Adaptation (AAAI 2025 · similarity 0.89)
- PACE: Marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization (NeurIPS 2024 · similarity 0.87)
- Parameter Efficient Fine-tuning via Explained Variance Adaptation (NeurIPS 2025 · similarity 0.86)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.86)
- Fine-Tuning is Fine, if Calibrated (NeurIPS 2024 · similarity 0.86)
- LEVI: Generalizable Fine-tuning via Layer-wise Ensemble of Different Views (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.