Research map / Label-efficient and robust learning / Zero-/few-shot and meta-learning
Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DIPA, a cross-domain few-shot method that fine-tunes only lightweight per-layer scale/shift parameters on a ViT (MIM-pretrained) using a proxy-anchor-style discriminative loss on the support set, reaching state-of-the-art on Meta-Dataset with ~0.3-0.5% tuned parameters.
Problem
Cross-domain few-shot classification: adapting pre-trained feature extractors to new classes in unseen domains with few labelled support samples, where full fine-tuning overfits and nearest centroid classifiers yield poorly separated clusters.
Approach
Replace full-backbone fine-tuning with a linear scale-and-shift transformation (gamma, beta) on LayerNorm, MLP and MHA outputs in the top layers only, and fine-tune these with a proxy-anchor loss that pulls samples to their hardest positives and pushes them from hardest negatives; also fuse [cls] tokens from the last several layers.
Key results
- Improves seen-domain accuracy by 7.7% over TSA in the SDL setting and unseen-domain accuracy by 5.3% over the compared methods reported in the paper's Table 2 (abstract claim).
- Under MDL, DIPA reaches 82.6% seen and 84.3% unseen average accuracy vs. TSA+SSA 80.8/79.0 (Table 2 rows A4 vs A6).
- Uses 0.3-0.5% of ViT-Small parameters during meta-testing (0.06-0.10M of 21.72M) versus up to 100% for PMF.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use the discriminative sample-aware loss for fine-tuning instead of NCCmean (ablation) | DIPA · Meta-Dataset MDL | accuracy: up to 2.8% improvement in seen domains and 1.6% in unseen domains | Helped |
| Use mean class centroids for inference instead of learned anchors (ablation) | DIPA with discriminative sample-aware fine-tuning · Meta-Dataset MDL | accuracy: 3.9% and 5.3% on seen and unseen domains | Helped |
| Increase feature fusion depth from 1 to 4 (ablation) | DIPA · Meta-Dataset MDL | accuracy: 82.9 -> 83.3 | Helped |
| Increase feature fusion depth to 8 or 12 (ablation) | DIPA · Meta-Dataset MDL | accuracy: 82.9 -> 82.6; 82.9 -> 82.1 | Hurt |
| Use MIM instead of DINO for pre-training (ablation) | DIPA · Meta-Dataset SDL-E pre-training + SDL | accuracy: 83.3 -> 84.0 | Helped |
| Use DIPA instead of PMF | Meta-Dataset SDL-E pre-training + SDL · Meta-Dataset | accuracy: 2.6% in seen domains, 4.4% in commonly unseen domains, and 6.5% across all domains | Helped |
| Use DIPA instead of ATTNSCALE | Meta-Dataset SDL-E pre-training + MDL · Meta-Dataset | accuracy: 2.9% higher performance in unseen domains and 2.7% in seen domains | Helped |
| Use DIPA instead of PMF | Meta-Dataset SDL-E pre-training + MDL · Meta-Dataset unseen domains | accuracy: 3.3% | Helped |
| Use DIPA instead of TSA + SSA | Meta-Dataset MDL · TrafficSign | accuracy: +6.4 | Helped |
| Use DIPA instead of TSA + SSA | Meta-Dataset MDL · MS-COCO | accuracy: +6.7 | Helped |
| Use DIPA instead of TSA + SSA | Meta-Dataset MDL · CIFAR-10 | accuracy: +4.5 | Helped |
| Use DIPA instead of TSA + SSA | Meta-Dataset MDL · CIFAR-100 | accuracy: +10.4 | Helped |
Surprising findings
- Tuning fewer layers (dt=7 or 9) often gives higher accuracy than tuning all 12 layers, indicating full-depth adaptation is not always beneficial.
Limitations
- A fixed linear transformation is applied to every layer rather than being layer-specific.
- Tuning depth is restricted to two values (dt=7 for seen, dt=9 for unseen) rather than per-dataset optimal depths.
- Pre-training uses masked image modelling with 4 A100 GPUs, and results use high-resolution 224x224 inputs that may not be directly comparable to 84x84 methods.
Experimental setup
Datasets: Meta-Dataset ImageNet
Benchmarks: Meta-Dataset
Metrics: mean accuracy 95% confidence interval
Baselines: Simple CNAPS URT FLUTE TSA TSA+SSA eTT ATTNSCALE PMF CTX ProtoNet
Builds on: Proxy Anchor loss Masked Autoencoders TSA eTT PMF
Closest papers by meaning
- Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification (ICLR 2023 · similarity 0.90)
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.89)
- MOKD: Cross-domain Finetuning for Few-shot Classification via Maximizing Optimized Kernel Dependence (ICML 2024 · similarity 0.89)
- Leveraging Normalization Layer in Adapters with Progressive Learning and Adaptive Distillation for Cross-Domain Few-Shot Learning (AAAI 2024 · similarity 0.88)
- Flatten Long-Range Loss Landscapes for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.88)
- The Devil is in Low-Level Features for Cross-Domain Few-Shot Segmentation (CVPR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.