Research map / Label-efficient and robust learning / Knowledge distillation

Boosting Residual Networks with Group Knowledge

AAAI 2024 · Label-efficient and robust learning › Knowledge distillation · approach: Patient consistent distillation · 5 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Group Knowledge based Training (GKT), which divides residual-network subnets into hierarchical groups via subnet-in-subnet sampling, aggregates group logits by EMA, and supervises each subnet with the neighboring larger group's knowledge, plus an inheriting exponential decay sampling rule favoring large subnets.

Problem

Subnet training of residual networks (stochastic depth, stimulative training) applies the same supervision to all subnets regardless of capacity, creating a large capacity gap and wasting knowledge produced by subnets during training.

Approach

Implicitly group subnets by hierarchical subnet-in-subnet sampling, keep per-group logit EMAs as group knowledge, and let each subnet be supervised by the next-larger group's knowledge with no extra teacher or main-net forward; larger subnets are sampled more via an exponential decay rule.

Key results

• On CIFAR-100, GKT reaches 84.38% Top-1 on WRN28-10 vs 82.17% baseline, and up to 1.54% over the second-best method.
• On ImageNet-1K, GKT improves ResNet-50 from 76.98% to 78.10% Top-1 at lower training cost (194.32 vs 202.16 GPU hours).
• On TinyImageNet, GKT improves ViT-S-like transformer results by up to 4.57% over baseline (per text), and reaches 68.13% on ResNet-34.

Measured effects reported in the paper

Surprising findings

• Kendall's Tau between all-subnet performance and main-net performance is 0, while large-subnet performance correlates at 0.67 with main-net performance, suggesting training large subnets matters more.

Limitations

• Gains on ImageNet-1K are modest (about 1.1 points for ResNet-50) and GKT(+epoch) is needed for further gains at higher cost.
• Hyperparameters (beta, alpha, M, q) are tuned on CIFAR-100 and reused across datasets; claimed insensitivity is shown only on WRN28-10 CIFAR-100.

Experimental setup

Datasets: CIFAR-10 CIFAR-100 Tiny ImageNet ImageNet-1K COCO2017

Benchmarks: CIFAR-100 Tiny ImageNet ImageNet-1K CIFAR-10 COCO2017

Metrics: Top-1 accuracy GPU hours memory mAP PQ/SQ/RQ sharpness gap Kendall's Tau

Baselines: Stochastic depth Stimulative training (ST) ONE CS-KD PS-KD LWR DLB

Builds on: stochastic depth stimulative training unraveled view of residual networks linear decay sampling sharpness gap capacity measure

Closest papers by meaning

• Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing (NeurIPS 2022 · similarity 0.87)
• Residual Alignment: Uncovering the Mechanisms of Residual Networks (NeurIPS 2023 · similarity 0.85)
• Distilling the Knowledge in Data Pruning (ICML 2025 · similarity 0.84)
• Deep Graph Neural Networks via Posteriori-Sampling-based Node-Adaptative Residual Module (NeurIPS 2024 · similarity 0.83)
• Efficient stagewise pretraining via progressive subnetworks (ICLR 2025 · similarity 0.83)
• S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning (NeurIPS 2025 · similarity 0.83)