Research map / Label-efficient and robust learning / Knowledge distillation
CIFD: Controlled Information Flow to Enhance Knowledge Distillation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CIFD, which replaces expensive teacher assistants with small rate-distortion modules operating on teacher embeddings and adds an information bottleneck module to the student, improving KD on CIFAR-100, ImageNet, and CLIP distillation on CC-12M.
Problem
Knowledge distillation suffers when the teacher is much larger than the student; prior teacher-assistant (TA) methods work but are expensive to train, since TAs must be trained from scratch, and overfitting arises as more assistants are added.
Approach
Replace TAs with Rate-Distortion Modules (RDMs): 2-3 layer bottleneck networks that take the frozen teacher's penultimate embedding and reconstruct it through a rate-constrained noisy channel, with the rate R setting the effective TA size. Also add an Information Bottleneck Module (IBM) at the student during training as a regularizer.
Key results
- On ImageNet, CIFD with 3 RDMs reaches 72.32% top-1 for ResNet34->ResNet18, +1.66 over KD (70.67%).
- On CIFAR-100 with simple CNNs, CIFD (5 RDM) reaches 47.25% vs 44.76% for KD and 46.05% for DGKD.
- On CLIP distillation (CC-12M), CIFD (1 RDM) with ViT-B-16 student reaches 54.10% ImageNet top-1 vs 51.25% for CLIPKD; training cost is about 1.08x KD versus 3.5x-4.5x for TA methods.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace KD with CIFD using 5 RDMs | 10-layer CNN → 2-layer CNN · CIFAR-100 | accuracy: 44.76 -> 47.25 | Helped |
| Use 3 RDMs instead of 1 RDM (ablation) | CIFD · CIFAR-100 | accuracy: 46.17 -> 46.68 | Helped |
| Use 5 RDMs instead of 3 RDMs (ablation) | CIFD with IBM · CIFAR-100 | accuracy: 46.68 -> 47.25 | Helped |
| Add IBM (ablation) | CIFD with 1 RDM · CIFAR-100 | accuracy: 45.34 -> 46.17 | Helped |
| Add IBM (ablation) | CIFD with 3 RDMs · CIFAR-100 | accuracy: 46.04 -> 46.68 | Helped |
| Add IBM (ablation) | CIFD with 5 RDMs · CIFAR-100 | accuracy: 45.14 -> 47.25 | Helped |
| Add IBM (ablation) | KD without RDM · CIFAR-100 | accuracy: 44.76 -> 45.68 | Helped |
| Add IBM (ablation) | CIFD with 1 RDM · ImageNet | Top-1 accuracy: 71.83 -> 72.05 | Helped |
| Add IBM (ablation) | CIFD with 3 RDMs · ImageNet | Top-1 accuracy: 72.22 -> 72.32 | Helped |
| Increase the number of RDMs from 1 to 3 (ablation) | CIFD · ImageNet zero-shot classification, ViT-S-16 | Top-1 accuracy: 49.37 -> 49.98 | Helped |
| Increase the number of RDMs from 1 to 3 (ablation) | CIFD · ObjectNet zero-shot classification, ViT-S-16 | Top-1 accuracy: 29.49 -> 29.37 | Hurt |
| Increase the number of RDMs from 1 to 3 (ablation) | CIFD · Flickr30k zero-shot retrieval, image-to-text | R@1: 66.90 -> 67.80 | Helped |
Surprising findings
- Increasing the number of RDMs without the IBM degrades CIFAR-100 accuracy at 5 RDMs (45.14% vs 47.25% with IBM), indicating overfitting that the IBM mitigates.
- Training the TinyCLIP distillation with ViT-B-16 teacher-student pairs diverged despite multiple attempts.
Limitations
- Future work noted: alternative schemes for sequentially enabling/disabling RDM feedback during training.
- Student depends on teacher, so teacher biases transfer to the student.
- Many baseline numbers are reproduced by the authors rather than taken from papers, making comparisons less direct.
Experimental setup
Datasets: CIFAR-100 ImageNet Conceptual Captions 12M
Benchmarks: CIFAR-100 ImageNet ImageNet-V2 ImageNet-A ImageNet-R ObjectNet COCO Flickr30k
Metrics: top-1 accuracy top-5 accuracy recall@K training cost (MACs normalized to KD)
Baselines: KD FT TAKD DGKD IFD NormKD IPWD DistKD CLIPKD TinyCLIP MobileCLIP OpenCLIP
Builds on: Knowledge Distillation (Hinton et al.) Teacher Assistant KD (TAKD) Densely Guided KD (DGKD) Information Bottleneck Principle CLIP-KD
Closest papers by meaning
- Flow-Based Knowledge Transfer for Efficient Large Model Distillation (AAAI 2026 · similarity 0.91)
- Heterogeneous Knowledge Distillation Using Information Flow Modeling (CVPR 2020 · similarity 0.90)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.87)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.87)
- Small Scale Data-Free Knowledge Distillation (CVPR 2024 · similarity 0.87)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.