Research map / Label-efficient and robust learning / Knowledge distillation
Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes DCKD, a structure-agnostic knowledge distillation framework for image restoration that uses a dynamically updated (EMA history model) negative-sample lower bound in contrastive regularization plus a VQGAN-codebook pixel-level category distribution alignment module.
Problem
Knowledge distillation for image restoration uses a fixed solution space and relies mainly on L1-type losses, which weakens lower-bound constraints as the student evolves and ignores image distribution information.
Approach
Dynamic contrastive regularization generates negatives from an EMA history model of the student applied to randomly degraded inputs, so the lower bound adapts to the student's learning state; a distribution mapping module aligns pixel-wise VQGAN codebook category distributions between teacher and student via cross-entropy.
Key results
- On SwinIR x4, DCKD* outperforms the SOTA MiPKD by 0.25 dB on Urban100 (per paper's claim of combined DCKD with DUKD).
- On Restormer deraining, DCKD outperforms Logits KD by 0.55 dB on Rain100L (38.02 vs 37.17 dB) with 3.8M params vs 26.1M teacher.
- On GoPro deblurring, DCKD improves NAFNet student to 31.43 dB, 0.17 dB over Logits KD, and Restormer student to 31.78 dB.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add DCR and DMM to the student distillation framework (ablation) | RCAN · Urban100 ×2 | PSNR: 32.67 -> 32.92 | Helped |
| Add DCR to a framework with DMM (ablation) | RCAN · Urban100 ×2 | PSNR: 32.81 -> 32.92 | Helped |
| Add DMM to a framework with DCR (ablation) | RCAN · Urban100 ×2 | PSNR: 32.83 -> 32.92 | Helped |
| Use random noise rather than random mixed degradation (ablation) | DCKD degradation module · Urban100 ×2 | PSNR: 32.87 -> 32.92 | Helped |
| Use DCKD* by combining DCKD with DUKD | SwinIR · Urban100 ×4 | PSNR: 26.53 -> 26.62 | Helped |
| Use DCKD* by combining DCKD with DUKD | RCAN · Urban100 ×4 | PSNR: 26.69 -> 26.74 | Helped |
| Distill with DCKD rather than Logits KD | NAFNet · GoPro | PSNR: 31.26 -> 31.43 | Helped |
| Distill with DCKD rather than Logits KD | Restormer · GoPro | PSNR: 31.61 -> 31.78 | Helped |
| Distill with DCKD rather than Logits KD | Restormer · Rain100L | PSNR: +0.55dB | Helped |
| Set λ_dcl to 0.1 rather than 0.01 (ablation) | DCKD on RCAN · Urban100 ×2 | PSNR: 32.87 -> 32.92 | Helped |
| Set λ_dcl to 1.0 rather than 0.1 (ablation) | DCKD on RCAN · Urban100 ×2 | PSNR: 32.92 -> 32.81 | Hurt |
| Set λ_ce to 0.001 rather than 0.0001 (ablation) | DCKD on RCAN · Urban100 ×2 | PSNR: 32.84 -> 32.92 | Helped |
Surprising findings
- Existing KD methods such as FAKD on SwinIR Urban100 x2 perform worse than training from scratch.
Limitations
- Gains over prior KD methods are modest (often around 0.1 dB).
- Hyperparameters (number of negatives, initial update step, lambda weights) are tuned via ablations on a single setting (RCAN x2, Set14/Urban100).
Experimental setup
Datasets: DIV2K GoPro Rain13k-style deraining collection (13,712 pairs) Set5 Set14 BSD100 Urban100 Test100 Rain100H Rain100L Test2800 Test1200
Benchmarks: Set5 Set14 BSD100 Urban100 GoPro Test100 Rain100H Rain100L Test2800 Test1200 Manga109
Metrics: PSNR SSIM
Baselines: Scratch Logits KD FAKD MiPKD MCLIR MT-RNN DMPHN MPRNet SPAIR
Builds on: CSD VQGAN Real-ESRGAN DUKD
Closest papers by meaning
- Knowledge Distillation with Multi-granularity Mixture of Priors for Image Super-Resolution (ICLR 2025 · similarity 0.87)
- Learning from History: Task-agnostic Model Contrastive Learning for Image Restoration (AAAI 2024 · similarity 0.86)
- Random Conditioning for Diffusion Model Compression with Distillation (CVPR 2025 · similarity 0.86)
- DeepKD: A Deeply Decoupled and Denoised Knowledge Distillation Trainer (NeurIPS 2025 · similarity 0.86)
- Active Data Curation Effectively Distills Large-Scale Multimodal Models (CVPR 2025 · similarity 0.85)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.