Research map / Label-efficient and robust learning / Knowledge distillation
De-confounded Data-free Knowledge Distillation for Handling Distribution Shifts
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KDCI, a causal-intervention plug-in for data-free knowledge distillation that uses a clustered confounder dictionary and backdoor adjustment to compensate for distribution shifts between substitution and original data.
Problem
Data-free knowledge distillation trains student models on substitute (synthetic or sampled) data whose quality and class proportions differ from the original training data, and this distribution shift confounds student learning and limits performance.
Approach
Model the distribution shift as a confounder in a customized causal graph, build a confounder dictionary by K-Means++ clustering of teacher prediction features with cluster proportions, and apply backdoor adjustment (approximated via NWGM) to compensate student predictions with an attention-weighted prior, so it can be plugged into existing DFKD methods.
Key results
- KDCI improves six DFKD baselines on CIFAR-10 and CIFAR-100, with gains up to 15.54% on CIFAR-100 for DeepInv (resnet-34 to resnet-18 setting).
- On Tiny-ImageNet, KDCI raises Fast from 28.79% to 38.23%, DeepInv from 20.68% to 34.84%, and DFND from 42.64% to 49.54% with negligible overhead (about 1-3% extra time).
- On ImageNet, KDCI improves DFND from 16.03% to 34.32% on resnet-50 to mobilenetv2 and Fast by up to 7.10%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add KDCI to DFND | DFND · CIFAR-10 | accuracy: 95.36 -> 95.44 | Helped |
| Add KDCI to DFND | DFND · CIFAR-100 | accuracy: 74.42 -> 77.09 | Helped |
| Add KDCI to DFND | DFND · Tiny-ImageNet | accuracy: 42.64 -> 49.54 | Helped |
| Add KDCI to DFND | DFND · ImageNet | accuracy: 42.82 -> 51.26 | Helped |
| Add KDCI to DFND | DFND · ImageNet, ResNet-50 → MobileNetV2 | accuracy: 16.03 -> 34.32 | Helped |
| Use learned attention weights instead of random λᵢ (ablation) | KDCI prior information · CIFAR-10 and CIFAR-100 | accuracy: 93.92 vs 94.56; 74.79 vs 75.10 (Fast); 91.56 vs 93.67; 72.72 vs 74.59 (DeepInv); 95.28 vs 95.38; 76.86 vs 77.09 (DFND) | Helped |
| Use extracted prototype confounders instead of randomly initialized zᵢ (ablation) | KDCI prior information · CIFAR-10 and CIFAR-100 | accuracy: 93.35 vs 94.56; 73.76 vs 75.10 (Fast); 91.84 vs 93.67; 72.81 vs 74.59 (DeepInv); 94.94 vs 95.38; 76.14 vs 77.09 (DFND) | Helped |
| Include prototype proportions P(zᵢ) (ablation) | KDCI prior information · CIFAR-10 and CIFAR-100 | accuracy: 93.70 vs 94.56; 74.60 vs 75.10 (Fast); 92.76 vs 93.67; 72.66 vs 74.59 (DeepInv); 95.11 vs 95.38; 76.97 vs 77.09 (DFND) | Helped |
| Use a confounder dictionary from the teacher trained on the other CIFAR dataset (ablation) | KDCI · CIFAR-10 and CIFAR-100 | accuracy: Fast: 94.56 vs 93.96; DeepInv: 93.67 vs 93.56; DFND: 95.44 vs 95.41 (CIFAR-10); Fast: 75.10 vs 74.75; DeepInv: 74.59 vs | Hurt |
| Add KDCI to Fast | Fast · Tiny-ImageNet | accuracy: 28.79 -> 38.23 | Helped |
| Add KDCI to DeepInv | DeepInv · Tiny-ImageNet | accuracy: 20.68 -> 34.84 | Helped |
| Add KDCI to Fast | Fast · ImageNet, ResNet-50 → ResNet-18 | accuracy: 53.45 -> 58.24 | Helped |
Surprising findings
- Using a confounder dictionary built from a different dataset's teacher (swapped CIFAR-10/CIFAR-100 teachers) reduces performance, which the authors attribute to over- and under-intervention.
- KDCI gains are small for Fast and CMI on CIFAR, which the authors attribute to their use of teacher BatchNorm statistics implicitly performing likelihood estimation.
Limitations
- Gains are smaller for generation-based methods trained with one generator per class on ImageNet, where the authors suspect the per-class generators already mitigate shifts.
- Performance depends on the confounder dictionary size N and the choice of pre-trained model used to build the confounder, requiring tuning.
- Approximates backdoor adjustment with NWGM rather than exact multi-pass computation, so the causal estimate is approximate.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet Flickr1M ImageNet unlabeled subset (600k)
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet
Metrics: top-1 accuracy FID GPU time per epoch memory usage
Baselines: DAFL Fast CMI DeepInv Mosaick DFND
Builds on: DAFL DeepInv CMI Fast DFND Mosaick
Closest papers by meaning
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.89)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.89)
- AuG-KD: Anchor-Based Mixup Generation for Out-of-Domain Knowledge Distillation (ICLR 2024 · similarity 0.88)
- DKDM: Data-Free Knowledge Distillation for Diffusion Models with Any Architecture (CVPR 2025 · similarity 0.87)
- What Makes a Good Dataset for Knowledge Distillation? (CVPR 2025 · similarity 0.86)
- Impartial Adversarial Distillation: Addressing Biased Data-Free Knowledge Distillation via Adaptive Constrained Optimization (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.