Research map / Architectures and efficiency / Attention and transformers
Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Cluster-Learngene, which density-clusters attention heads and FFNs of a pre-trained DeiT ancestry model into learngene centroids and inherits them, with priority weight-sharing and learnable FFN transforms, to initialize elastic-scale ViT descendants.
Problem
Pre-trained ViTs are one-size-fits-all and ill-suited to task-specific resource constraints; existing Learngene methods extract whole layers and stack them, retaining redundant weights and lacking elastic scaling.
Approach
Adaptively cluster attention heads per layer using mean attention distance as a density metric (DBSCAN-style, no preset cluster count), keep the closest-to-centroid head as the learngene, merge FFNs across adjacent layers with similar head centroids, and expand centroids to any head count via priority weight-sharing and learnable FFN transformations.
Key results
- On DeiT-Small with 6 layers, Cluster-Learngene reaches 71.09% on iNat-2019 (11.17 points above Auto-Learngene) and 85.38% on CIFAR-100, outperforming Pretraining-Finetuning on six of seven datasets.
- On ImageNet-1K, a Small-scale descendant (Hd=6, Ld=12) reaches 78.43% versus 64.91% from scratch at 100 epochs, and the paper reports about 2x reduced training cost.
- With 25% of ImageNet-1K training data, Base-scale descendants reach 72.82% versus 65.73% for From-Scratch (+7.09%), at roughly 1/6 the training cost.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · iNat-2019 | Top-1 accuracy (%): 68.48 -> 71.09 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · Food-101 | Top-1 accuracy (%): 87.8 -> 89.53 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · Flowers | Top-1 accuracy (%): 91.13 -> 92.31 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · Cars | Top-1 accuracy (%): 86.81 -> 89.87 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · CIFAR-10 | Top-1 accuracy (%): 97.59 -> 97.79 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · CIFAR-100 | Top-1 accuracy (%): 84.43 -> 85.38 | Helped |
| initialize the DeiT-Small descendant with Cluster-Learngene instead of Pretraining-Finetuning | DeiT-Small · CUB-200 | Top-1 accuracy (%): 78.13 -> 75.98 | Hurt |
| use adaptive clustering rather than k-means clustering (ablation) | Cluster-Learngene · CIFAR-100 | accuracy (%): 80.12 -> 85.38; 81.02 -> 85.38; 81.41 -> 85.38 | Helped |
| use priority weight-sharing instead of sharing heads in the original sequence (ablation) | Cluster-Learngene · CIFAR-100 | accuracy (%): 83.98 -> 85.38 | Helped |
| use priority weight-sharing instead of sharing heads in increasing order of ˜d (ablation) | Cluster-Learngene · CIFAR-100 | accuracy (%): 84.24 -> 85.38 | Helped |
| use priority weight-sharing instead of sharing heads in decreasing order of ˜d (ablation) | Cluster-Learngene · CIFAR-100 | accuracy (%): 84.63 -> 85.38 | Helped |
| select cluster-nearest head and FFN modules instead of averaging parameters (ablation) | Cluster-Learngene · CIFAR-100 | accuracy (%): 85.12 -> 85.38 | Helped |
Surprising findings
- Cluster-Learngene does not beat From-Scratch on the full ImageNet-1K training set for Base-scale models (81.80 vs 78.43 for Small-scale reference in Table 3 setting), its advantage appears mainly with limited data.
- Incrementing attention heads with depth outperforms decrementing by about 1.45% accuracy on average, contrary to a uniform allocation default.
Limitations
- Descendant quality depends heavily on the quality of the ancestry model; biases in the ancestry model can propagate to descendants.
- Clustering hyperparameters (Eps=10, MinHds=1, epsilon=10) were chosen empirically and the ablations are mainly on CIFAR-100 with DeiT-Small.
Experimental setup
Datasets: ImageNet-1K iNaturalist-2019 Food-101 Oxford Flowers Stanford Cars CIFAR-10 CIFAR-100 CUB-200-2011
Benchmarks: ImageNet-1K iNaturalist-2019 Food-101 Oxford Flowers Stanford Cars CIFAR-10 CIFAR-100 CUB-200-2011
Metrics: Top-1 accuracy FLOPs parameters training speedup
Baselines: Pretraining-Finetuning From-Scratch Heuristic-Learngene Weight-Transformation Auto-Learngene k-means clustering
Builds on: Learngene DeiT DBSCAN Auto-Learngene Heuristic-Learngene
Closest papers by meaning
- Vision Transformers as Probabilistic Expansion from Learngene (ICML 2024 · similarity 0.90)
- Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation (NeurIPS 2024 · similarity 0.89)
- Transformer as Linear Expansion of Learngene (AAAI 2024 · similarity 0.87)
- Inheriting Generalized Learngene for Efficient Knowledge Transfer across Multiple Tasks (AAAI 2025 · similarity 0.86)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.85)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.