Research map / Label-efficient and robust learning / Knowledge distillation

Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models

NeurIPS 2026 · Label-efficient and robust learning › Knowledge distillation · approach: Rethinking distillation objectives · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes CaRE-KD, a knowledge distillation framework for LLMs that gates between forward and reverse KL per token by confidence (CaRE-Divergence) and rejects batches where the teacher is epistemically less reliable than the student via MC-dropout BALD (Revival).

Problem

Standard forward-KL knowledge distillation assumes a reliable teacher, but LLM teachers have high-entropy predictions and hallucinations, so matching their full distribution degrades well-calibrated student priors.

Approach

A per-token confidence-gated mix of Forward KL and Reverse KL (driven by normalized-entropy confidence gap), plus a sequence-level rejection mask that skips updates when the teacher's BALD uncertainty is high relative to the student's running quantile.

Key results

• CaRE-Divergence improves average ROUGE-L over Forward KL by +2.6 on GPT2-base (SGO setting, Table 1).
• On MBPP, CaRE-KD reaches 62.4 pass@1, +2.1 over SRKL; GSM8K 73.9% (+1.7 over SRKL).
• Revival applied to CaRE-Divergence gives +1.42 average ROUGE-L on OPT-125M; reported overhead ~42% at N=3 on Dolly with OpenLLaMA-3B.

Surprising findings

• Revival applied to Forward KL on average reduces performance (FKL delta -3.91 on Gemma-2-2B-IT), and Reverse KL gains are not reliably positive.
• On OpenLLaMA-7B to 3B, CaRE-KD is statistically tied with the best static baseline, as the teacher is strong and the compression ratio small.

Limitations

• BALD via MC dropout adds substantial overhead, up to ~170% on code per the authors' per-domain breakdown.
• The Sharpness Hypothesis underlying the entropy-BALD proxy is assumed and weakly supported; main results use full MC-dropout BALD.
• Gains are mixed: CaRE-KD is not best on every pair, and the teacher-factuality link to Revival is not statistically significant.

Experimental setup

Datasets: Dolly-15k UltraChat-200k WizardCoder MetaMathQA

Benchmarks: Dolly Evaluation Self-Instruct Super-Natural Instructions Vicuna Instructions AlpacaEval Evol-Instruct UltraFeedback HumanEval MBPP GSM8K CollegeMath

Metrics: ROUGE-L LLM-as-a-judge factuality pass@1 exact-match accuracy

Baselines: FKL RKL SRKL alpha-beta divergence SFT KD SeqKD ImitKD MiniLLM GKD Distillm ABKD

Builds on: MiniLLM Skewed KL alpha-beta divergence GKD BALD LoRA

Closest papers by meaning

• DistiLLM: Towards Streamlined Distillation for Large Language Models (ICML 2024 · similarity 0.92)
• Entropy-Aware On-Policy Distillation of Language Models (ICML 2026 · similarity 0.91)
• TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models (ICLR 2025 · similarity 0.91)
• ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.91)
• Strong Teacher Not Needed? On Distillation in LLM Pretraining (NeurIPS 2026 · similarity 0.91)
• WASD: Wasserstein-based Knowledge Distillation for Large Language Models (NeurIPS 2026 · similarity 0.91)