Research map / Label-efficient and robust learning / Knowledge distillation
A Knowledge Distillation-Based Approach to Enhance Transparency of Classifier Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Distills a DenseNet121 teacher into a five-layer CNN student via KD and visualizes layer-averaged feature maps for interpretability on three medical image datasets.
Problem
Deep CNNs used for medical image classification are large and opaque, so clinicians cannot easily understand their decision process; the paper asks how to build a simple, accurate, and interpretable classifier.
Approach
Use knowledge distillation from a DenseNet121 teacher to a shallow five-layer student CNN, then interpret the student by averaging feature maps per layer (KD-FMV), which yields faster explanations than Grad-CAM and SHAP on the teacher.
Key results
- Best student on brain tumor reaches 0.9748 test accuracy vs 0.9877 for the DenseNet121 teacher (table).
- Best eye-disease student reaches 0.9351 accuracy vs 0.9837 for the teacher, roughly a 5% drop.
- Student FLOPs are 232.70M vs 566.89M for the teacher, and SHAP mean execution time falls from about 67.7 s to about 15.6 s per image.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| distill DenseNet121 into the five-layer student CNN using α=0.7, T=10 | KD-FMV · Brain tumor test | accuracy: 0.9877 -> 0.9748 | Hurt |
| distill DenseNet121 into the five-layer student CNN using α=0.4, T=15 | KD-FMV · Eyes disease test | accuracy: 0.9837 -> 0.9351 | Hurt |
| distill DenseNet121 into the five-layer student CNN using α=0.4, T=5 | KD-FMV · Alzheimer test | accuracy: 0.9938 -> 0.9946 | Helped |
| vary α and T across the reported settings (ablation) | KD-FMV student · Brain tumor test | accuracy: 0.9572; 0.9656; 0.9748; 0.9702; 0.9748; 0.9717 | Mixed |
| vary α and T across the reported settings (ablation) | KD-FMV student · Eyes disease test | accuracy: 0.9074; 0.8750; 0.9351; 0.8958; 0.8680; 0.9219 | Mixed |
| vary α and T across the reported settings (ablation) | KD-FMV student · Alzheimer test | accuracy: 0.9946; 0.9906; 0.9928; 0.9934; 0.9915; 0.9919 | Mixed |
| use the five-layer student instead of the teacher for Grad-CAM analysis | KD-FMV · Brain tumor | Fidelity score (Avg): 0.9298 -> 0.9277 | Hurt |
| use the five-layer student instead of the teacher for Grad-CAM analysis | KD-FMV · Eyes disease | Fidelity score (Avg): 0.9059 -> 0.8630 | Hurt |
| use the five-layer student instead of the teacher for Grad-CAM analysis | KD-FMV · Alzheimer | Fidelity score (Avg): 0.9112 -> 0.9132 | Helped |
| use the five-layer student instead of the teacher for SHAP analysis | KD-FMV · Brain tumor | Fidelity score (Avg): 0.9404 -> 0.9277 | Hurt |
| use the five-layer student instead of the teacher for SHAP analysis | KD-FMV · Eyes disease | Fidelity score (Avg): 0.8700 -> 0.8630 | No clear effect |
| use the five-layer student instead of the teacher for SHAP analysis | KD-FMV · Alzheimer | Fidelity score (Avg): 0.9359 -> 0.9132 | Hurt |
Surprising findings
- On Alzheimer's data the student (0.9946 accuracy) slightly outperforms the teacher (0.9938).
- Grad-CAM/SHAP fidelity scores on the teacher are not uniformly higher than the proposed student-based explanations; the proposed method is lower on several classes.
Limitations
- Accuracy drops about 5% on the eye disease dataset, which the authors attribute to shallow networks.
- Interpretability is assessed largely qualitatively, and the fidelity metric as defined uses adversarial samples in a way that is weakly specified.
- The student's explanations are compared against teacher-based Grad-CAM and SHAP rather than clinician ground truth.
Experimental setup
Datasets: Brain tumor MRI dataset (figshare, SARTAJ, Br35H) Eye disease fundus dataset Alzheimer MRI dataset
Benchmarks: Brain tumor MRI dataset Eye disease fundus dataset Alzheimer MRI dataset
Metrics: accuracy F1 score loss fidelity score FLOPs mean execution time
Baselines: Grad-CAM SHAP DenseNet121
Builds on: knowledge distillation Grad-CAM SHAP
Closest papers by meaning
- Class Attention Transfer Based Knowledge Distillation (CVPR 2023 · similarity 0.86)
- VkD: Improving Knowledge Distillation using Orthogonal Projections (CVPR 2024 · similarity 0.85)
- Search to Distill: Pearls Are Everywhere but Not the Eyes (CVPR 2020 · similarity 0.85)
- Decoupled Knowledge Distillation (CVPR 2022 · similarity 0.85)
- MICA: Towards Explainable Skin Lesion Diagnosis via Multi-Level Image-Concept Alignment (AAAI 2024 · similarity 0.84)
- Debiased Distillation for Consistency Regularization (AAAI 2025 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.