Research map / Architectures and efficiency / RNNs and normalization
Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add LatentBE with weight-averaged subnetworks to KD | WRN28x4 distilled from DE-4 · CIFAR-100 | ACC: 79.09±0.20 -> 79.46±0.20 | Helped |
| Add TDiv-SDiv perturbation to LatentBE (ablation) | WRN28x4 distilled from DE-4 · CIFAR-100 | ACC: 79.46±0.20 -> 80.02±0.07 | Helped |
| Use ConfODS rather than no input perturbation (ablation) | LatentBE-4 · CIFAR-100 | ACC: 79.46±0.20 -> 79.27±0.30 | Hurt |
| Increase teacher diversity only in the perturbation (ablation) | LatentBE-4 · CIFAR-100 | ACC: 79.46±0.20 -> 79.40±0.03 | No clear effect |
| Include decreasing student diversity alongside increasing teacher diversity (ablation) | LatentBE-4 · CIFAR-100 | NLL: 0.993±0.024 -> 0.792±0.004 | Helped |
| Add TDiv-SDiv perturbation to LatentBE (ablation) | WRN28x1 distilled from DE-4 · CIFAR-10 | NLL: 0.263±0.003 -> 0.205±0.006 | Helped |
| Add TDiv-SDiv perturbation to LatentBE (ablation) | WRN28x4 distilled from DE-4 · CIFAR-100-C | ACC: 50.03±0.36 -> 51.36±0.28 | Helped |
| Add TDiv-SDiv perturbation to LatentBE (ablation) | R18 distilled from DE-4 · TinyImageNet | ACC: 68.96±0.19 -> 69.14±0.27 | Helped |
| Add TDiv-SDiv perturbation to LatentBE (ablation) | R50 distilled from DE-2 · ImageNet-1k | ACC: 77.30 -> 77.38 | Helped |
| Average the trained rank-one factors for single-network inference (ablation) | LatentBE · CIFAR-100 | NLL | Helped |
| Use student-aware TDiv-SDiv perturbation instead of ConfODS (ablation) | LatentBE · CIFAR-10 | NLL: 0.223±0.007 -> 0.205±0.006 | Helped |
| Distill with LatentBE and TDiv-SDiv rather than vanilla KD | WRN28x4 distilled from DE-4 · CIFAR-100 | inference time: 0.107 ms/img -> 0.107 ms/img | No clear effect |
Closest papers by meaning
- Diversity Matters When Learning From Ensembles (NeurIPS 2021 · similarity 0.91)
- Ensemble Distribution Distillation via Flow Matching (ICML 2025 · similarity 0.89)
- The Diversified Ensemble Neural Network (NeurIPS 2020 · similarity 0.89)
- MEAL: Multi-Model Ensemble via Adversarial Learning (AAAI 2019 · similarity 0.89)
- Provably Near-Optimal Federated Ensemble Distillation with Negligible Overhead (ICML 2025 · similarity 0.86)
- United We Stand: Using Epoch-Wise Agreement of Ensembles to Combat Overfit (AAAI 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.