Research map / Label-efficient and robust learning / Knowledge distillation
Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Kdiffusion, a feature-level knowledge distillation method that decomposes the teacher-student transfer into progressive diffusion-style reverse steps, using structural re-parameterization to generate multiple student features that are merged linearly at inference.
Problem
Feature-level knowledge distillation with L2 loss assumes Gaussian outputs and ignores variance, which fails under large teacher-student distribution gaps (e.g., Transformer-to-CNN), causing negative transfer.
Approach
Decompose the transfer objective into intermediate diffusion reverse steps optimized progressively, and avoid the costly diffusion model by generating multiple student features via structural re-parameterization that combine linearly at inference with no extra cost; add target-guided training and shuffle sampling.
Key results
- On CIFAR-100 (ResNet32x4 to ShuffleNetV1), Kdiffusion with multiple-layer features reaches 77.90 vs. 75.11 for CRD and 77.45 for Review's baseline comparison at 70.50 student accuracy.
- On ImageNet-100 with Swin teacher, Kdiffusion (multi-layer) reaches 86.20 on MobileNetV2 (vs. 85.00 KD) and 80.68 on ShuffleNetV2 (vs. 80.02 DKD).
- On ImageNet-1k (ResNet50 to MobileNetV2), Kdiffusion2 achieves 73.62 top-1 vs. 71.37 for DKD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add target-guided training to multiple-layer Kdiffusion (ablation) | Kdiffusion · CIFAR-100, ResNet32x4 teacher and ShuffleNetV1 student | accuracy: 76.83 -> 77.26 | Helped |
| Use multiple-layer instead of single-layer Kdiffusion (ablation) | Kdiffusion · CIFAR-100, ResNet32x4 teacher and ShuffleNetV1 student | accuracy: 75.83 -> 76.83 | Helped |
| Use multiple-layer instead of single-layer Kdiffusion (ablation) | Kdiffusion · CIFAR-100, ResNet50 teacher and MobileNetV2 student | accuracy: 69.20 -> 69.95 | Helped |
| Use multiple-layer instead of single-layer Kdiffusion (ablation) | Kdiffusion · ImageNet-100, Swin teacher and ResNet18 student | accuracy: 86.18 -> 86.30 | Helped |
| Use multiple-layer instead of single-layer Kdiffusion (ablation) | Kdiffusion · ImageNet-1k, ResNet50 teacher and MobileNet student | Top-1 accuracy: 73.48 -> 73.62 | Helped |
| Use multiple-layer instead of single-layer Kdiffusion (ablation) | Kdiffusion · ImageNet-1k, ResNet34 teacher and ResNet18 student | Top-1 accuracy: 71.68 -> 72.04 | Helped |
| Add extra re-parameterized student features at stage 1 without teacher supervision (ablation) | MobileNetV2 · CIFAR-100 | accuracy: 64.60 -> 64.22 | Hurt |
| Add extra re-parameterized student features at stages 1 and 2 without teacher supervision (ablation) | MobileNetV2 · CIFAR-100 | accuracy: 64.60 -> 63.32 | Hurt |
| Add extra re-parameterized student features at stages 1, 2 and 3 without teacher supervision (ablation) | MobileNetV2 · CIFAR-100 | accuracy: 64.60 -> 62.31 | Hurt |
| Add extra re-parameterized student features at all four stages without teacher supervision (ablation) | MobileNetV2 · CIFAR-100 | accuracy: 64.60 -> 61.96 | Hurt |
| Increase re-parameterized student feature count (ablation) | Kdiffusion, ShuffleNetV1 · CIFAR-100, ResNet32x4 teacher | accuracy: 74.85 -> 75.96 -> 76.28 -> 76.80 | Helped |
| Increase re-parameterized student feature count (ablation) | Kdiffusion, MobileNetV2 · CIFAR-100, ResNet50 teacher | accuracy: 67.87 -> 68.46 -> 68.91 -> 70.16 | Helped |
Surprising findings
- Conventional feature-level methods such as CRD give inferior or negative gains in the Transformer-to-CNN setting, while logits-level KD and DKD remain stable.
- Simply using uniform average weights for the re-parameterized student features performs worse than the proposed shuffle-sampling scheme, indicating the features do not automatically share learning load.
Limitations
- Training cost increases with the number of re-parameterized student features, limited by memory.
- Uniform-weight combination for training is not practical, requiring shuffle sampling workaround.
- Re-parameterization applied only to the last layer of each stage for fair comparison, so full potential not explored.
Experimental setup
Datasets: CIFAR-100 ImageNet-100 ImageNet-1k
Benchmarks: CIFAR-100 ImageNet-100 ImageNet-1k
Metrics: top-1 accuracy top-5 accuracy
Baselines: KD DKD FitNet PKT RKD CRD AT VID OFD Review SP
Builds on: FitNet Review CRD DKD Structural re-parameterization (RepVGG-style, Ding et al. 2021)
Closest papers by meaning
- Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teacher (ICLR 2025 · similarity 0.91)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.89)
- Unlocking Dataset Distillation with Diffusion Models (NeurIPS 2025 · similarity 0.88)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.88)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.88)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.