Research map / Detection and segmentation / CNN features
Inheriting Generalized Learngene for Efficient Knowledge Transfer across Multiple Tasks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CLKG, a two-stage Learngene method that distills a large-kernel CNN (RepLKNet-XL) into a small-kernel auxiliary model, selects convolutional kernels as learngene, and inherits them to initialize architecturally diverse descendant CNNs for classification, detection, and segmentation.
Problem
Learngene methods extract compact knowledge from a large pretrained model to initialize smaller descendants, but existing ones do not ensure the extracted knowledge generalizes across tasks and architectures; KD suffers from negative transfer and needs repeated distillation for each student architec
Approach
Extract convolutional kernels that carry generalized feature-extraction ability from a small-kernel auxiliary model distilled densely from a large-kernel CNN, selecting kernels via direct extraction, edge-priority and continuity criteria, then adapt their width to initialize backbones of different CNN architectures.
Key results
- CLKG improves over training from scratch by +2.89% on VOC12+SBD (DeepLabv3p) and +2.59% on COCO2017 (YOLOV7), and +3.64% on miniImageNet (VGG16).
- CLKG reduces training data volume and epochs by roughly 2x while matching or beating scratch training.
- On negative-transfer tasks, FT (feature distillation) drops 0.81% below scratch on Oxford Flower while CLKG gains +2.83%; CLKG gains +1.88% on NAO.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use CLKG learngene initialization instead of random initialization | VGG16, BotNet, ResNet50, YOLOV7, DETR, DeepLabv3p, and Mobile-ViT · Table 1 downstream benchmarks | performance: +3.64%; +2.63%; +2.83%; +2.65%; +2.59%; +1.88%; +2.36%; +1.92%; +1.53%; +2.89%; +1.53%; +2.11%; +2.95% | Helped |
| Use CLKG rather than feature distillation (FT) | VGG16 · Oxford Flower | performance: 86.19 -> 85.38; CLKG 89.02 | Helped |
| Use CLKG rather than feature distillation (FT) | YOLOV7 · NAO | performance: 15.22 -> 14.54; CLKG 17.10 | Helped |
| Use CLKG rather than feature distillation (FT) | DeepLabv3p · mini ADE20K | performance: 56.59 -> 55.77; CLKG 58.12 | Helped |
| Distill outputs from all three stages instead of only the last stage (ablation) | CLKG · VGG16 | performance: 74.67 -> 76.69 | Helped |
| Distill outputs from all three stages instead of the first and last stages (ablation) | CLKG · VGG16 | performance: 75.86 -> 76.69 | Helped |
| Distill outputs from all three stages instead of only the last stage (ablation) | CLKG · YOLOV7 | performance: 81.08 -> 82.81 | Helped |
| Use Swin-T rather than RepLKNet-XL as ancestry model (ablation) | CLKG · VGG16 | performance: 76.69 -> 75.83 | Hurt |
| Use Swin-T rather than RepLKNet-XL as ancestry model (ablation) | CLKG · YOLOV7 | performance: 82.81 -> 82.44 | Hurt |
| Use Swin-T rather than RepLKNet-XL as ancestry model (ablation) | CLKG · DeepLabv3p | performance: 57.85 -> 57.25 | Hurt |
Surprising findings
- Feature distillation (FT) underperforms training from scratch on Oxford Flower, NAO, and mini ADE20K, showing negative transfer.
- Swin-T guidance gives smaller gains than RepLKNet-XL guidance for descendant CNNs, suggesting large-kernel ancestors transfer better to small-kernel students.
Limitations
- Restricted to CNN descendants; pure Vision Transformer architectures are not addressed.
- Evaluation limited to vision tasks and a single large-kernel ancestry model (RepLKNet-XL).
- Negative transfer comparisons cover only a few tasks against a limited set of KD baselines.
Experimental setup
Datasets: ImageNet-1K miniImageNet CIFAR-100 Oxford Flower VOC07+12 COCO2017 NAO VOC12+SBD mini-ADE20K
Benchmarks: miniImageNet CIFAR-100 Oxford Flower VOC07+12 COCO2017 NAO VOC12+SBD mini-ADE20K
Metrics: accuracy mAP mIoU
Baselines: Scratch Heur FT OKD-MTL DASFD
Builds on: Learngene RepLKNet knowledge distillation Heur FitNets-style dense feature distillation
Closest papers by meaning
- Cluster-Learngene: Inheriting Adaptive Clusters for Vision Transformers (NeurIPS 2024 · similarity 0.86)
- Initializing Variable-sized Vision Transformers from Learngene with Learnable Transformation (NeurIPS 2024 · similarity 0.85)
- Vision Transformers as Probabilistic Expansion from Learngene (ICML 2024 · similarity 0.85)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.84)
- Building Variable-Sized Models via Learngene Pool (AAAI 2024 · similarity 0.83)
- KIND: Knowledge Integration and Diversion for Training Decomposable Models (ICML 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.