Research map / Label-efficient and robust learning / Domain adaptation and generalization
Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Extends loss-landscape flatness analysis to representation space and proposes FLoR, a normalization layer that randomly interpolates BN and IN (or LN and BN for ViTs) representations to flatten long-range minima for cross-domain few-shot learning.
Problem
Cross-domain few-shot learning must transfer knowledge across dissimilar domains and fine-tune with very limited target data; prior flatness methods (SAM-style) only flatten short-range parameter-space landscapes and generalize poorly under large domain shifts.
Approach
Analyze loss landscapes in representation space rather than parameter space, treat differently normalized representations (BN, IN, LN) as distinct minima, and flatten the high-loss region between them by randomly interpolating these normalized representations at every layer via a drop-in FLoR normalization layer.
Key results
- FLoR with ResNet10 and fine-tuning reaches 52.67% average 5-way 1-shot accuracy and 66.22% average 5-way 5-shot accuracy under the transductive fine-tuning setting, the top average among compared methods.
- On ISIC2018 5-shot, FLoR outperforms prior state of the art by up to about 9%.
- With ViT-S (DINO), FLoR achieves 74.75% average 5-way 5-shot accuracy with fine-tuning, above StyleAdv's 74.06%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use FLoR instead of the BN baseline (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 57.26 ±0.13 -> 61.52 ±0.12 | Helped |
| Apply SAM to the representation-space loss landscape rather than the parameter-space loss landscape (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 57.47 ±0.38 -> 57.95 ±0.17 | Helped |
| Interpolate normalized representations using Mixup (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 57.26 ±0.13 -> 58.46 ±0.15 | Helped |
| Use Manifold Mixup (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 57.26 ±0.13 -> 58.53 ±0.15 | Helped |
| Replace BN with IN (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 57.26 ±0.13 -> 58.72 ±0.10 | Helped |
| Add input perturbation to the IN model (ablation) | ResNet10 with IN · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 58.72 ±0.10 -> 58.92 ±0.11 | Helped |
| Add input adversarial perturbation to the IN model (ablation) | ResNet10 with IN · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 58.72 ±0.10 -> 59.23 ±0.12 | Helped |
| Add representation perturbation to the IN model (ablation) | ResNet10 with IN · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 58.72 ±0.10 -> 59.66 ±0.10 | Helped |
| Use Meta BIN instead of FLoR (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 61.52 ±0.12 vs 59.13 ±0.12 | Hurt |
| Fix the mixing ratio for BN + IN (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 61.52 ±0.12 vs 59.59 ±0.13 | Hurt |
| Use separate BN and IN streams (ablation) | ResNet10 · CDFSL average, 5-way 5-shot (plain prototype classifier) | accuracy: 61.52 ±0.12 vs 59.66 ±0.14 | Hurt |
| Finetune all layers with a larger learning rate (ablation) | FLoR · CDFSL few-shot finetuning | accuracy | Helped |
Surprising findings
- Directly applying SAM on representations yields only marginal CDFSL gains, while the IN-based model tolerates larger perturbation step sizes.
Limitations
- Gains on some datasets are modest or negative relative to baselines (e.g., Places and Cars in some settings).
- Analysis of representation-space landscapes relies on perturbation proxies and sanity-check experiments rather than full landscape visualization.
Experimental setup
Datasets: miniImageNet CUB Cars Plantae Places CropDiseases EuroSAT ISIC2018 ChestX ImageNet-C
Benchmarks: CUB Cars Places Plantae ChestX ISIC2018 EuroSAT CropDiseases
Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy average accuracy
Baselines: GNN+FT GNN+ATA MN+AFA GNN+AFA LDP-net Fine-tuning TPN+ATA TPN+AFA RDC StyleAdv PMF Mixup
Builds on: SAM Instance Normalization Batch Normalization StyleAdv Meta BIN
Closest papers by meaning
- The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation (CVPR 2025 · similarity 0.89)
- A Closer Look at the CLS Token for Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.89)
- Random Registers for Cross-Domain Few-Shot Learning (ICML 2025 · similarity 0.88)
- Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes (CVPR 2025 · similarity 0.88)
- Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.88)
- Attention Temperature Matters in ViT-Based Cross-Domain Few-Shot Learning (NeurIPS 2024 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.