Research map / Label-efficient and robust learning / Knowledge distillation
Diversity Matters When Learning From Ensembles
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distill from the ensemble using ODS-perturbed training inputs | BatchEns-4 · CIFAR-100, WRN-28x10 | accuracy: 80.51\spm{0.22} -> 81.88\spm{0.32} | Helped |
| Distill from the ensemble using ODS-perturbed training inputs | BatchEns-4 · CIFAR-100, WRN-28x10 | DEE: 1.000\spm{0.007} -> 3.987\spm{1.325} | Helped |
| Distill from the ensemble using ODS-perturbed training inputs | BatchEns-4 · CIFAR-10, ResNet-32 | DEE: 1.261\spm{0.074} -> 1.938\spm{0.067} | Helped |
| Replace ODS perturbations with Gaussian noise (ablation) | BatchEns students distilled from DeepEns · CIFAR-10, ResNet-32 | DEE: 2.391\spm{0.113} -> 2.206\spm{0.175} | Hurt |
| Scale ODS steps according to teacher confidence (ablation) | BatchEns students distilled from DeepEns · CIFAR-10, ResNet-32 | DEE: 2.486\spm{0.164} -> 2.524\spm{0.080} | Helped |
| Use adversarial perturbations instead of ODS perturbations (ablation) | ResNet-32 student distilled from an ensemble · CIFAR-10 | accuracy: 93.98\spm{0.20} -> 93.32\pm0.22 | Hurt |
| Use adversarial perturbations instead of ODS perturbations (ablation) | ResNet-32 student distilled from an ensemble · CIFAR-10 | NLL: 0.188\spm{0.003} -> 0.194\pm0.003 | Hurt |
| Distill with ODS rather than Gaussian perturbations (ablation) | BatchEns-4 · CIFAR-100, WRN-28x2 student | accuracy: 74.50\spm{0.17} -> 76.03\spm{0.22} | Helped |
| Distill with ODS rather than Gaussian perturbations (ablation) | BatchEns-4 · CIFAR-100, WRN-28x5 student | accuracy: 78.80\spm{0.41} -> 80.24\spm{0.05} | Helped |
| Use ODS perturbations rather than no perturbation (ablation) | MIMO-3 student distilled from DeepEns-3 · CIFAR-100, WRN-28x10 | accuracy: 80.75 -> 81.17 | Helped |
| Use ODS perturbations rather than Gaussian perturbations (ablation) | MIMO-3 student distilled from DeepEns-3 · CIFAR-100, WRN-28x10 | accuracy: 80.69 -> 81.17 | Helped |
| Use ODS perturbations rather than Gaussian perturbations (ablation) | BatchEns-4 · CIFAR-10-C, ResNet-32 | NLL: 1.487\spm{0.026} -> 1.406\spm{0.044} | Helped |
Closest papers by meaning
- Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (ICML 2022 · similarity 0.91)
- Boosting Ensemble Accuracy by Revisiting Ensemble Diversity Metrics (CVPR 2021 · similarity 0.90)
- DIBS: Diversity Inducing Information Bottleneck in Model Ensembles (AAAI 2021 · similarity 0.90)
- Sharpness-diversity tradeoff: improving flat ensembles with SharpBalance (NeurIPS 2024 · similarity 0.89)
- On the Diversity of Adversarial Ensemble Learning (ICML 2025 · similarity 0.88)
- Ensemble Distribution Distillation via Flow Matching (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.