Research map / Label-efficient and robust learning / Knowledge distillation
In Good GRACES: Principled Teacher Selection for Knowledge Distillation
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham M. Kakade, Surbhi Goel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use GRACE to rank teachers instead of gradient norm or G-Vendi | teacher selection · GSM8K (LLaMA-1B) | Spearman correlation: 0.55 -> 0.86 vs gradient norm; 0.44 -> 0.86 vs G-Vendi | Improved |
| use GRACE instead of gradient norm for choosing teacher | teacher selection · GSM8K (LLaMA-1B) | regret: 4.9% -> 0.3% | Improved |
| select teacher by teacher accuracy | teacher selection · GSM8K (LLaMA-1B) | Spearman correlation: 11% | Hurt |
| select teacher by student's loss on teacher generations | teacher selection · GSM8K (LLaMA-1B) | Spearman correlation: 44%; regret 5.4% | Hurt |
| use GRACE to choose generation temperature | teacher temperature · GSM8K (LLaMA-1B) | predicted optimal temperature: 0.8 vs predicted 0.9 (3B); 0.4 vs 0.5 (1.5B) | Improved |
| use GRACE within a teacher size budget | teacher selection · GSM8K (LLaMA-1B) | regret: below 0.3% vs at least 9% | Improved |
| increase cross-validation splits above 6 (ablation) | GRACE · GSM8K and MATH | correlation | No clear effect |
| vary number of prompts, generations, and projection dimension (ablation) | GRACE · GSM8K (LLaMA-1B) | correlation | No clear effect |
| filter teacher generations for correctness (ablation) | distillation data · GSM8K | student performance | No clear effect |
| measure by greedy or best-of-k instead of average-at-16 | GRACE · GSM8K | Spearman correlation: regret <2.5% | Hurt |
| use GRACE on simple OOD MATH Perturb vs gradient norm | teacher selection · MATH Perturb (Simple) | correlation | Hurt |
| use GRACE on hard OOD MATH settings | teacher selection · MATH^2 and MATH Perturb (Hard) | correlation and regret | Improved |
Closest papers by meaning
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.89)
- Toward Student-oriented Teacher Network Training for Knowledge Distillation (ICLR 2024 · similarity 0.89)
- Can Students Beyond the Teacher? Distilling Knowledge from Teacher’s Bias (AAAI 2025 · similarity 0.88)
- Teacher as a Lenient Expert: Teacher-Agnostic Data-Free Knowledge Distillation (AAAI 2024 · similarity 0.88)
- The Signal is in the Steps: Local Scoring for Reasoning Data Selection (ICML 2026 · similarity 0.88)
- Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
