AI Paper Atlas

Research map / Label-efficient and robust learning / Knowledge distillation

In Good GRACES: Principled Teacher Selection for Knowledge Distillation

Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Sham M. Kakade, Surbhi Goel

ICLR 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 6 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use GRACE to rank teachers instead of gradient norm or G-Venditeacher selection · GSM8K (LLaMA-1B)Spearman correlation: 0.55 -> 0.86 vs gradient norm; 0.44 -> 0.86 vs G-VendiImproved
use GRACE instead of gradient norm for choosing teacherteacher selection · GSM8K (LLaMA-1B)regret: 4.9% -> 0.3%Improved
select teacher by teacher accuracyteacher selection · GSM8K (LLaMA-1B)Spearman correlation: 11%Hurt
select teacher by student's loss on teacher generationsteacher selection · GSM8K (LLaMA-1B)Spearman correlation: 44%; regret 5.4%Hurt
use GRACE to choose generation temperatureteacher temperature · GSM8K (LLaMA-1B)predicted optimal temperature: 0.8 vs predicted 0.9 (3B); 0.4 vs 0.5 (1.5B)Improved
use GRACE within a teacher size budgetteacher selection · GSM8K (LLaMA-1B)regret: below 0.3% vs at least 9%Improved
increase cross-validation splits above 6 (ablation)GRACE · GSM8K and MATHcorrelationNo clear effect
vary number of prompts, generations, and projection dimension (ablation)GRACE · GSM8K (LLaMA-1B)correlationNo clear effect
filter teacher generations for correctness (ablation)distillation data · GSM8Kstudent performanceNo clear effect
measure by greedy or best-of-k instead of average-at-16GRACE · GSM8KSpearman correlation: regret <2.5%Hurt
use GRACE on simple OOD MATH Perturb vs gradient normteacher selection · MATH Perturb (Simple)correlationHurt
use GRACE on hard OOD MATH settingsteacher selection · MATH^2 and MATH Perturb (Hard)correlation and regretImproved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy