Research map / Label-efficient and robust learning / Knowledge distillation
Cooperative Knowledge Distillation: A Learner Agnostic Approach
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes cooperative distillation, where multiple trained models act as both teachers and students by exchanging quintessential counterfactual instances for the instances each model gets wrong but another gets right.
Problem
Standard knowledge distillation transfers all knowledge unidirectionally from one teacher to one student, regardless of whether that knowledge is useful; it does not allow multiple trained models to cooperatively fill each other's specific deficiencies.
Approach
Each model identifies instances it predicts correctly and others get wrong, then generates quintessential counterfactuals (pushing the teacher's prediction toward the true class by alpha=0.5) that are added to the student's training set; this is learner-agnostic and works across architectures, algorithms, and partially overlapping feature spaces.
Key results
- In Experiment 1, the Craigslist MLP improves from 60.98% to 68.68% and the Auction Export MLP from 86.04% to 87.74%, outperforming all listed competitors.
- In Experiment 4, heterogeneous models (decision tree, naive Bayes, SVM) on heart disease data improve to 66.24%, 77.56%, and 59.08% from baselines of 56.71%, 62.37%, and 54.45%.
- In the 10-CNN FashionMNIST stress test, median accuracy rises from 76% to 83% (baseline 86% topline), while knowledge distillation pretraining (86%) is the only competitor that beats it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Apply cooperative distillation | CL MLP · Car Guru test, Exp. 1 | accuracy: 60.98% -> 68.68% | Helped |
| Apply cooperative distillation | AE MLP · Car Guru test, Exp. 1 | accuracy: 86.04% -> 87.74% | Helped |
| Apply cooperative distillation | CL decision tree · Car Guru test, Exp. 2 | accuracy: 63.41% -> 67.58% | Helped |
| Apply cooperative distillation | AE MLP · Car Guru test, Exp. 2 | accuracy: 86.04% -> 87.32% | Helped |
| Apply cooperative distillation | Model 1 decision tree · Cleveland test, Exp. 3 | accuracy: 56.71% -> 57.44% | Helped |
| Apply cooperative distillation | Model 2 decision tree · Cleveland test, Exp. 3 | accuracy: 43.45% -> 62.36% | Helped |
| Apply cooperative distillation | Model 3 decision tree · Cleveland test, Exp. 3 | accuracy: 53.47% -> 63.04% | Helped |
| Apply cooperative distillation | Model 1 decision tree · Cleveland test, Exp. 4 | accuracy: 56.71% -> 66.24% | Helped |
| Apply cooperative distillation | Model 2 Naive Bayes · Cleveland test, Exp. 4 | accuracy: 62.37% -> 77.56% | Helped |
| Apply cooperative distillation | Model 3 SVM · Cleveland test, Exp. 4 | accuracy: 54.45% -> 59.08% | Helped |
| Remove similar counterfactuals using geometric set-cover (ablation) | Cooperative distillation · FashionMNIST, median across 10 models | accuracy: 79% -> 83% | Helped |
| Use cooperative distillation instead of KD as pretraining | CNN models · FashionMNIST, median across 10 models | accuracy: 86% -> 83% | Hurt |
Surprising findings
- A low-performance model (Craigslist MLP) can teach a high-performance model and improve it, which the authors say no other distillation technique replicated.
- In the FashionMNIST stress test, knowledge distillation as pretraining outperformed the proposed method.
Limitations
- Relies on counterfactual generation quality, which is harder for complex data such as images.
- Works best when training data distributions differ significantly and features overlap.
- Pairwise exchange cost grows with the number of model permutations (P(k,2)).
Experimental setup
Datasets: Craigslist used car data Auction Export used car data Car Guru test set Heart disease datasets (Long Beach, Switzerland, Hungary, Cleveland) FashionMNIST Statlog German Credit
Benchmarks: Craigslist/Auction Export/Car Guru used car price Heart disease (Cleveland test set) FashionMNIST Statlog German Credit
Metrics: test accuracy median accuracy over runs
Baselines: parameter transfer DCGAN TabGAN mixup response-based KD feature-based KD KD as pretraining add training data together
Builds on: offline knowledge distillation counterfactual explanations self-distillation DCGAN mixup
Closest papers by meaning
- Learning Student-Friendly Teacher Networks for Knowledge Distillation (NeurIPS 2021 · similarity 0.92)
- What Knowledge Gets Distilled in Knowledge Distillation? (NeurIPS 2023 · similarity 0.91)
- Does Knowledge Distillation Really Work? (NeurIPS 2021 · similarity 0.90)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.89)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.89)
- Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.