Research map / Label-efficient and robust learning / Domain adaptation and generalization
Few-shot Adaptation to Distribution Shifts By Mixing Source and Target Embeddings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MixPro, which trains a linear probe on a large dataset formed by fixed-weight linear mixing of source embeddings with same-class few-shot target embeddings, for data-efficient few-shot adaptation to distribution shifts.
Problem
Adapting pretrained models to distribution shifts when only a few (2-16 per class) labeled target examples are available; last-layer retraining (DFR) overfits and Pro² may discard target-relevant features learned only from source data.
Approach
Construct a large mixed-embedding dataset by linearly combining each source embedding with a random same-class target embedding using a fixed weight s, then train a linear probe on it, so the probe sees both source and target information without overfitting the few target examples.
Key results
- MixPro outperforms baselines by up to 7% with only 2-4 target examples per class across 8 datasets.
- On average MixPro outperforms baselines by 4.3% (2-shot) and 3.9% (4-shot).
- With hyperparameters chosen by 2-fold cross-validation on the few target examples, MixPro remains the best method, with performance drops mostly within 10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Mix source and target embeddings before linear probing | Few-shot adaptation with MixPro · 8 distribution-shift datasets, 2-shot | accuracy: 4.3% | Helped |
| Mix source and target embeddings before linear probing | Few-shot adaptation with MixPro · 8 distribution-shift datasets, 4-shot | accuracy: 3.9% | Helped |
| Use MixPro instead of existing baselines | Few-shot adaptation · 8 distribution-shift datasets | accuracy: up to 7% | Helped |
| Use only target data for linear probing instead of mixing source data | Few-shot adaptation · Small target datasets | accuracy | Hurt |
| Use a source-learned projection before target linear probing | Pro² · Severe distribution shifts, including domain generalization datasets | accuracy | Hurt |
| Tune hyperparameters with few-shot cross-validation instead of a large validation set | Few-shot adaptation methods · Terra Incognita, VLCS, UrbanCars, bFFHQ, Office-Home | accuracy: within 10% | Hurt |
| Replace source embeddings with source class-mean embeddings (ablation) | MixPro · 8 distribution-shift datasets | accuracy | Hurt |
| Use an intermediate source-target mixing weight instead of using only source or only target data (ablation) | MixPro · Synthetic subpopulation shift | test loss | Helped |
| Increase the target-data mixing weight as source-target shift severity increases (ablation) | MixPro · Synthetic subpopulation shift | test loss | Helped |
| Decrease the target-data mixing weight as noise or scarcity of target examples increases (ablation) | MixPro · Synthetic subpopulation shift | test loss | Helped |
Surprising findings
- Sampling the mixing weight from a Beta distribution (as in Mixup) gives expected equal source/target weighting, which can be suboptimal.
- Pro² performs worse than other methods when the target contains minority subpopulations only or has examples absent from the source support.
Limitations
- MixPro requires source embeddings to be stored and available simultaneously with target data; MixPro-CM with class means performs slightly worse.
- Theoretical analysis relies on synthetic Gaussian models and high-dimensional asymptotics.
- Mixing weight s must be selected; few-shot cross-validation slightly degrades performance.
Experimental setup
Datasets: Waterbirds UrbanCars bFFHQ Camelyon17 PACS VLCS Office-Home Terra Incognita
Benchmarks: Waterbirds UrbanCars bFFHQ Camelyon17 PACS VLCS Office-Home Terra Incognita
Metrics: test accuracy
Baselines: DFR Mixup Manifold Mixup Pro² diversity-based multi-model method (gradient-similarity ensemble)
Builds on: DFR Pro² Mixup Manifold Mixup
Closest papers by meaning
- Selective Mixup Helps with Distribution Shifts, But Not (Only) because of Mixup (ICML 2024 · similarity 0.86)
- Leveraging Normalization Layer in Adapters with Progressive Learning and Adaptive Distillation for Cross-Domain Few-Shot Learning (AAAI 2024 · similarity 0.85)
- Discriminative Sample-Guided and Parameter-Efficient Feature Space Adaptation for Cross-Domain Few-Shot Learning (CVPR 2024 · similarity 0.85)
- Pushing the Limit of Fine-Tuning for Few-Shot Learning: Where Feature Reusing Meets Cross-Scale Attention (AAAI 2024 · similarity 0.84)
- BECLR: Batch Enhanced Contrastive Few-Shot Learning (ICLR 2024 · similarity 0.84)
- Provably Improving Generalization of Few-shot models with Synthetic Data (ICML 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.