Research map / Label-efficient and robust learning / Knowledge distillation
Ensemble Distribution Distillation via Flow Matching
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EDFM, an ensemble distribution distillation method that uses conditional flow matching in logit space with a lightweight MLP student to transfer the diversity of an ensemble teacher.
Problem
Ensemble distillation into a single cheap student leaves a performance gap to the ensemble teacher, partly because teacher predictions on training data are highly correlated and student capacity is limited; existing methods fail to transfer teacher diversity.
Approach
Model the distribution of ensemble teacher logits given an input with rectified flow matching in logit space, conditioned on pretrained teacher features, using a small denoising MLP; sampling multiple logits gives diverse student predictions cheaply and in parallel.
Key results
- On CIFAR-100, EDFM reaches NLL 0.932 and ECE 0.056 versus 0.965 NLL for KD and 1.031 for EnDD, with ACC 0.761 vs Multi-SWAG teacher 0.777 and NLL 0.835.
- On CIFAR-10, EDFM achieves NLL 0.216 and ECE 0.009, the best NLL among distillation baselines (DBN 0.218, KD 0.229) with ACC 0.931.
- On commonsense reasoning with LLaMA-2-7B, EDFM reaches ARC-E ACC 0.892 and NLL 0.390, outperforming KD (0.850, 0.570) and near Multi-IVON teacher NLL 0.357.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use EDFM instead of KD | ResNet-32 ensemble distillation · CIFAR-100 test | NLL: 0.965±0.008 -> 0.932±0.003 | Helped |
| Use EDFM instead of KD | LLaMA-2-7B ensemble distillation · ARC-C | ACC: 0.662±0.032 -> 0.715±0.014 | Helped |
| Use EDFM instead of KD | LLaMA-2-7B ensemble distillation · ARC-E | NLL: 0.570±0.019 -> 0.390±0.002 | Helped |
| Use EDFM instead of KD | LLaMA-2-7B ensemble distillation · OBQA | ACC: 0.776±0.005 -> 0.818±0.010 | Helped |
| Distill with mixup, RandAugment, or auxiliary data to increase teacher diversity | EnDD and FED · CIFAR-10 | NLL | Mixed |
| Distill with greater teacher diversity using the tested diversification setups | EDFM · CIFAR-10 | NLL | Helped |
| Use EDFM instead of other distillation baselines | CIFAR-10 test predictions · CIFAR-10 test | Wasserstein-2 distance to teacher | Helped |
| Use EDFM for distribution-shift robustness | ResNet-32 ensemble distillation · CIFAR-10.1 | NLL | Helped |
| Use EDFM for distribution-shift robustness | ResNet-32 ensemble distillation · CIFAR-10.2 | NLL | Helped |
| Use EDFM for OOD detection | ResNet-32 ensemble distillation · SVHN OOD | AUROC | Helped |
| Use EDFM sampling instead of Multi-SWAG ensemble inference | CIFAR-100 setup · CIFAR-100 | NLL | Mixed |
| Use a simple rectified-flow formulation rather than more complex flow formulations (ablation) | EDFM · Image classification | performance | Helped |
Surprising findings
- Existing ensemble distribution distillation methods (EnDD, FED) do not consistently improve performance as teacher diversity increases, even though their student diversity grows.
Limitations
- Uses a simple rectified-flow formulation; more sophisticated simplex-constrained or diffusion variants were not explored.
- Hyperparameters such as sigma=4 and time distribution were selected through ablations on the CIFAR setup, and gains on LLM tasks are reported on a few commonsense benchmarks only.
- Multi-SWAG and Multi-IVON teachers are the only teachers evaluated; the ensemble-student performance gap to the teacher remains for CIFAR-100 accuracy.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 CINIC-10 ARC-C ARC-E OBQA
Benchmarks: CIFAR-10 CIFAR-100 CIFAR-10.1 CIFAR-10.2 STL SVHN ARC-C ARC-E OBQA
Metrics: ACC NLL ECE AUROC TNR95 TNR99 VAR AMB DGR TVD
Baselines: KD EnDD FED DBN Multi-SWAG teacher IVON baseline
Builds on: Flow Matching Diffusion Schrödinger Bridge distillation (DBN) Ensemble distribution distillation (EnDD) FED Multi-SWAG Multi-IVON
Closest papers by meaning
- Improving Ensemble Distillation With Weight Averaging and Diversifying Perturbation (ICML 2022 · similarity 0.89)
- Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space (NeurIPS 2020 · similarity 0.88)
- How to Trade Off the Quantity and Capacity of Teacher Ensemble: Learning Categorical Distribution to Stochastically Employ a Teacher for Distillation (AAAI 2024 · similarity 0.88)
- Diversity Matters When Learning From Ensembles (NeurIPS 2021 · similarity 0.88)
- Progressively Knowledge Distillation via Re-parameterizing Diffusion Reverse Process (AAAI 2024 · similarity 0.85)
- Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.