Research map / Optimization / Optimizers
Task-Specific Preconditioner for Cross-Domain Few-Shot Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Task-Specific Preconditioned gradient descent (TSP), which meta-learns positive-definite domain-specific preconditioners and combines them with dataset-classifier task-coefficients to adapt task-specific parameters on Meta-Dataset.
Problem
Cross-domain few-shot learning adapts task-specific parameters with fixed optimizers (GD or fixed preconditioners), which is sub-optimal across domains and target tasks.
Approach
Meta-learn one positive-definite preconditioner (M^T M + I) per meta-training domain, then form a task-specific preconditioner as a convex combination weighted by task-coefficients predicted by a dataset classifier, and apply it in preconditioned gradient descent on the task-specific parameters.
Key results
- In the multi-domain setting, TSP applied on TA^2-Net (TSP††) reaches 80.1→81.4 average all-domain accuracy and best average rank 1.0, vs. 80.1 for TA^2-Net.
- In the single-domain setting, TSP applied on TA^2-Net reaches 75.7 average accuracy across all datasets (76.9 unseen), the best among compared methods.
- Ablation: the Gram-matrix DSP (M^T M + I) with PD constraint yields 80.9 average all vs. 77.6 without the PD constraint.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply TSP to TSA | TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 79.5 -> 80.9 | Helped |
| Apply TSP to TA²-Net | TA²-Net · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 80.1 -> 81.4 | Helped |
| Apply TSP to TSA | TSA · Meta-Dataset single-domain Varying-Way Varying-Shot | Avg All accuracy: 73.3 -> 75.0 | Helped |
| Apply TSP to TA²-Net | TA²-Net · Meta-Dataset single-domain Varying-Way Varying-Shot | Avg All accuracy: 73.8 -> 75.7 | Helped |
| Use the MᵀM+I DSP design instead of LLᵀ (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 80.1 -> 80.9 | Helped |
| Use the MᵀM+I DSP design instead of LLᵀ+I (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 80.5 -> 80.9 | Helped |
| Add a positive-definite constraint to the preconditioner (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 77.6 -> 80.9 | Helped |
| Set the dataset-classifier loss weighting factor to λ=0.1 (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 80.7 -> 80.9 | Helped |
| Use MᵀM+I instead of MᵀM for the DSP design (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot | Avg All accuracy: 75.6 -> 80.9 | Helped |
| Use MᵀM+I instead of MᵀM for the DSP design (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Five-Shot | Avg All accuracy: 71.7 -> 76.3 | Helped |
| Add the identity matrix to the LLᵀ DSP design (ablation) | TSP on TSA · Meta-Dataset multi-domain Varying-Way Varying-Shot and Varying-Way Five-Shot | Avg All accuracy: 80.1 -> 80.5; 75.0 -> 74.6 | Mixed |
Surprising findings
- Without an explicit PD constraint, roughly 24-35% of DSPs lose positive definiteness after meta-training even when initialized as 0.1*I.
- Adding the identity matrix to the DSP helps in the varying-way varying-shot setting but hurts in the five-shot setting.
Limitations
- Gains over the base methods are moderate (e.g., roughly 1-2 points on average), and the method is built on top of existing TSA/TA^2-Net pipelines.
- Non-PD preconditioners without a constraint can fail to converge, so the PD constraint design is essential and restricts the parameterization.
Experimental setup
Datasets: Meta-Dataset ILSVRC 2012 Omniglot FGVC-Aircraft CUB-200-2011 Describable Textures QuickDraw FGVCx Fungi VGG Flower Traffic Signs MSCOCO MNIST
Benchmarks: Meta-Dataset
Metrics: mean accuracy 95% confidence interval average rank average seen/unseen/all accuracy
Baselines: SUR URT FLUTE tri-M URL TSA TA^2-Net MOKD BOHB ALFA+Proto-MAML GAP+Proto-MAML Simple-CNAPS
Builds on: TSA TA^2-Net GAP Meta-Dataset URL
Closest papers by meaning
- On The Concurrence of Layer-wise Preconditioning Methods and Provable Feature Learning (ICML 2025 · similarity 0.86)
- MetaDiff: Meta-Learning with Conditional Diffusion for Few-Shot Learning (AAAI 2024 · similarity 0.85)
- Can Looped Transformers Learn to Implement Multi-step Gradient Descent for In-context Learning? (ICML 2024 · similarity 0.85)
- Optimization Inspired Few-Shot Adaptation for Large Language Models (NeurIPS 2025 · similarity 0.85)
- Task-Specific Gradient Adaptation for Few-Shot One-Class Classification (CVPR 2025 · similarity 0.85)
- Linear Transformers are Versatile In-Context Learners (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.