Research map / Label-efficient and robust learning / Recognition and long-tail
Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
placeholder
Problem
Data-free meta-learning from a pool of pre-trained models suffers performance degradation because heterogeneous models create conflicting tasks; prior DFML methods ignore this heterogeneity.
Approach
Group the most dissimilar pre-trained models together (task dissimilarity from FIM task embeddings of model-inverted data, clustered by spectral clustering), then apply an implicit gradient regularization (approximated via a displaced-gradient trick) within each group to align conflicting task gradients, framing it as a heterogeneity-homogeneity trade-off.
Key results
- Outperforms the strongest prior DFML baseline BiDf-MKD by 5.92%, 5.54% and 2.25% on 1-shot CIFAR-FS, miniImageNet and CUB, and by 6.33%, 6.70% and 3.05% in 5-shot.
- In the multi-domain scenario, improves over the baseline by 7.19% (1-shot) and 16.53% (5-shot) per the paper's text.
- Number of task groupings c=5 best among tested values on CIFAR-FS 5-way 5-shot (49.00), with performance peaking near 6-7 groups.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · CIFAR-FS 5-way 1-shot | accuracy: 37.66 ± 0.75 -> 43.58 ± 0.82 | Helped |
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · CIFAR-FS 5-way 5-shot | accuracy: 51.16 ± 0.79 -> 57.49 ± 0.79 | Helped |
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · miniImageNet 5-way 1-shot | accuracy: 30.66 ± 0.59 -> 36.20 ± 0.71 | Helped |
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · miniImageNet 5-way 5-shot | accuracy: 42.30 ± 0.64 -> 49.00 ± 0.71 | Helped |
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · CUB 5-way 1-shot | accuracy: 31.62 ± 0.60 -> 33.87 ± 0.63 | Helped |
| Use Task Groupings Regularization instead of BiDf-MKD | DFML meta-learning · CUB 5-way 5-shot | accuracy: 44.32 ± 0.69 -> 47.37 ± 0.73 | Helped |
| Add task grouping to vanilla meta-model optimization (ablation) | Task Groupings Regularization · CIFAR-FS 5-way 1-shot | accuracy: 41.84 ± 0.82 -> 43.06 ± 0.81 | Helped |
| Add IGR to task-grouped training (ablation) | Task Groupings Regularization · CIFAR-FS 5-way 5-shot | accuracy: 55.84 ± 0.76 -> 57.49 ± 0.79 | Helped |
| Add task grouping to IGR-only training (ablation) | Task Groupings Regularization · CUB 5-way 5-shot | accuracy: 47.92 ± 0.69 -> 47.37 ± 0.73 | Hurt |
| Add classification loss L_CE to task recovery (ablation) | Task Groupings Regularization · CIFAR-FS 5-way 1-shot | accuracy: 32.36 ± 0.59 -> 43.58 ± 0.82 | Helped |
| Add batch-normalization feature regularization L_BN (ablation) | Task Groupings Regularization · CIFAR-FS 5-way 5-shot | accuracy: 54.71 ± 0.76 -> 57.49 ± 0.79 | Helped |
| Add hard-label supervision (ablation) | Task Groupings Regularization · CUB 5-way 5-shot | accuracy: 47.72 ± 0.70 -> 47.37 ± 0.73 | Hurt |
Surprising findings
- Grouping dissimilar pre-trained models outperforms grouping similar ones (e.g., 57.49 vs 55.71 on CIFAR-FS 5-shot), counter to the common multi-task intuition.
- Model heterogeneity can be beneficial, with combining a dissimilar auxiliary model sometimes raising accuracy as a regularizer, while 100% overlapping or identical architectures yield little gain.
Limitations
- Number of task groups must be set manually (c=5); automatic selection left to future work
- Evaluated only on image few-shot classification with Conv4-style meta-models and small datasets
- Gradient regularization relies on a small-learning-rate approximation whose theory holds only for small finite steps
Experimental setup
Datasets: CIFAR-FS miniImageNet CUB
Benchmarks: CIFAR-FS miniImageNet CUB
Metrics: 5-way 1-shot accuracy 5-way 5-shot accuracy
Baselines: Finetune Best-Model Average OTA DRO PURER BiDf-MKD
Builds on: PURER BiDf-MKD MAML Implicit gradient regularization (Smith et al.)
Closest papers by meaning
- Efficient and Effective Multi-task Grouping via Meta Learning on Task Combinations (NeurIPS 2022 · similarity 0.90)
- DMTG: One-Shot Differentiable Multi-Task Grouping (ICML 2024 · similarity 0.88)
- FairMT: Fairness for Heterogeneous Multi-Task Learning (NeurIPS 2026 · similarity 0.87)
- Towards Principled Task Grouping for Multi-Task Learning (NeurIPS 2026 · similarity 0.87)
- Learning to Learn with Contrastive Meta-Objective (NeurIPS 2025 · similarity 0.87)
- Any-Way Meta Learning (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.