Research map / Label-efficient and robust learning / Knowledge distillation
Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MiPKD, a knowledge distillation framework for image super-resolution that mixes teacher and student features in a shared latent space (feature prior mixer) and stochastically swaps network blocks between teacher and student (block prior mixer).
Problem
Knowledge distillation for SR models is tailored to specific teacher-student architectures (depth-only or width-only compression) and fails to generalize, especially under compounded depth+width compression, because feature maps from teacher and student have different semantics and capacity.
Approach
Randomly mixes encoded teacher and student feature maps via a random 3D mask in a unified latent space (feature prior mixer), then builds stochastic sub-networks that route the fused features through teacher or student blocks (block prior mixer), supervised by teacher outputs and ground truth.
Key results
- On EDSR x2/x3/x4, MiPKD improves Urban100 PSNR over the scratch student by 0.56 dB, 0.42 dB, and 0.24 dB respectively.
- On RCAN depth distillation, MiPKD improves Urban100 PSNR over scratch by 0.2 dB, 0.17 dB, and 0.19 dB at x2, x3, x4.
- In the high-compression EDSR x4 setting (c64b16), MiPKD+MiPKD two-stage distillation reaches 25.89 dB on Urban100 versus 25.61 dB for no-KD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MiPKD instead of CSD | EDSR c64b32 · Urban100 ×2, ×3, and ×4 | PSNR: 32.26 -> 32.52; 28.21 -> 28.41; 26.34 -> 26.45 | Helped |
| Use MiPKD instead of FAKD | RCAN c64b6 · Urban100 ×2, ×3, and ×4 | PSNR: 32.65 -> 32.83; 28.62 -> 28.76; 26.50 -> 26.55 | Helped |
| Use MiPKD instead of logits KD | SwinIR · Urban100 ×2, ×3, and ×4 | PSNR: 32.09 -> 32.46; 28.23 -> 28.47; 26.15 -> 26.37 | Helped |
| Use MiPKD for both distillation stages | EDSR c64b16 · Urban100 ×4, compounded compression | PSNR: 25.79 -> 25.89 | Helped |
| Add the feature prior mixer (ablation) | MiPKD on RCAN c32b5g5 · Urban100 | PSNR: 25.60 -> 25.63 | Helped |
| Add the block prior mixer to the feature prior mixer (ablation) | MiPKD on RCAN c32b5g5 · Urban100 | PSNR: 25.63 -> 25.69 | Helped |
| Use separate teacher and student encoders instead of a shared encoder (ablation) | MiPKD feature mixer · Urban100 | PSNR: 25.61 -> 25.63 | Helped |
| Add encoders to map features to the latent space (ablation) | MiPKD feature mixer · Urban100 | PSNR: 24.51 -> 25.63 | Helped |
| Use convolutional rather than MLP encoder/decoder (ablation) | MiPKD on RCAN c64b5g10 · Urban100 | PSNR: 26.42 -> 26.55 | Helped |
| Use a random mask rather than Cosine, grid, or CKA masking (ablation) | MiPKD feature prior mixture · Urban100 | PSNR: 25.62 -> 25.69; 25.61 -> 25.69; 25.63 -> 25.69 | Helped |
| Add the auxiliary auto-encoder loss (ablation) | MiPKD on RCAN · Urban100 | PSNR: 26.42 -> 26.55 | Helped |
| Set λblock to 0.1 while λfeat is 1 (ablation) | MiPKD on RCAN · Urban100 | PSNR: 26.51 -> 26.55 | Helped |
Surprising findings
- Existing feature-based KD methods (AT, FitNet) underperform the vanilla student trained without KD in most settings.
- Directly distilling into the highly compressed student degrades performance, requiring a two-stage teacher-assistant pipeline.
Limitations
- Gains are modest in absolute terms (often under 0.6 dB PSNR).
- Several baselines (CSD x2/x3) were reproduced by the authors since checkpoints were unavailable, and some compression settings are only partially compared.
Experimental setup
Datasets: DIV2K
Benchmarks: Set5 Set14 BSD100 Urban100
Metrics: PSNR SSIM FLOPs FPS
Baselines: Scratch Logits-KD RKD AT FitNet FAKD CSD
Builds on: FitNet FAKD CSD RKD AT
Closest papers by meaning
- AugKD: Ingenious Augmentations Empower Knowledge Distillation for Image Super-Resolution (ICLR 2025 · similarity 0.88)
- Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration (AAAI 2025 · similarity 0.87)
- Over-parameterized Student Model via Tensor Decomposition Boosted Knowledge Distillation (NeurIPS 2024 · similarity 0.86)
- All You Need in Knowledge Distillation Is a Tailored Coordinate System (AAAI 2025 · similarity 0.86)
- Neural Collapse Inspired Knowledge Distillation (AAAI 2025 · similarity 0.86)
- Bayesian Knowledge Distillation: A Bayesian Perspective of Distillation with Uncertainty Quantification (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.