Research map / Label-efficient and robust learning / Knowledge distillation
Few-Shot Knowledge Distillation of LLMs With Counterfactual Explanations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes CoD, which augments few-shot task-aware LLM distillation with LLM-generated counterfactual explanations (flip-validated by the teacher), using half the original samples plus their CFEs, with theoretical motivation via Fisher information and Hausdorff distance.
Problem
Task-aware knowledge distillation of LLMs usually needs large datasets, but in many practical settings only a handful of labeled samples are available; few-shot distillation remains underexplored.
Approach
Pair each few-shot training input with a counterfactual explanation (a minimally perturbed input that flips the teacher's prediction), generated by GPT-4o and validated by the teacher, so the student better matches the teacher's decision boundary; supported by a logistic-regression Fisher information result and a Hausdorff-distance bound.
Key results
- On Amazon Polarity with k=8, KD+CoD reaches 75.8% vs 67.1% for KD.
- On IMDB with k=8, LWD+CoD reaches 86.1% vs 76.0% for LWD, a gain of more than 10 points.
- With the DeBERTa-v3 setup, CoD gains are largest at small k (k<=64) and mostly vanish by k=512, where some datasets slightly underperform the baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add counterfactual explanations to LWD training | DeBERTa-v3-base → DeBERTa-v3-small · IMDB k=8 | accuracy: 0.760\ts{0.046} -> 0.861\ts{0.017} | Helped |
| Add counterfactual explanations to KD training | DeBERTa-v3-base → DeBERTa-v3-small · Amazon Polarity k=8 | accuracy: 0.671\ts{0.046} -> 0.758\ts{0.027} | Helped |
| Add counterfactual explanations to KD or LWD training | DeBERTa-v3-base → DeBERTa-v3-small · CoLA, all reported k values | accuracy | Helped |
| Add counterfactual explanations to distillation | DeBERTa-v3-base → DeBERTa-v3-small · IMDB and SST2 k=512 | accuracy | Hurt |
| Remove teacher soft-label supervision (ablation) | CoD · Few-shot classification, all shot levels | accuracy | Hurt |
| Replace teacher soft labels with random values (ablation) | CoD · Few-shot classification | accuracy | Hurt |
| Vary the CFE-generation prompt template (ablation) | CoD · Few-shot classification | accuracy | No clear effect |
Surprising findings
- TED does not consistently outperform simpler KD or LWD in few-shot settings.
- CoD gains slightly underperform the baselines at k=512 on IMDB and SST2, possibly due to redundancy.
Limitations
- CFE generation via prompting LLMs does not guarantee the closest counterfactual, limiting precision.
- Generating CFEs adds computational overhead over standard distillation.
- Student quality depends on teacher quality; teacher boundary errors are inherited.
Experimental setup
Datasets: SST2 Sentiment140 IMDB CoLA Amazon Polarity Yelp
Benchmarks: SST2 Sentiment140 IMDB CoLA Amazon Polarity Yelp
Metrics: accuracy
Baselines: KD LWD TED
Builds on: knowledge distillation layer-wise distillation TED counterfactual explanations
Closest papers by meaning
- Distilling What Matters: Confidence-Aware Selective Distillation for Large Language Models (NeurIPS 2026 · similarity 0.90)
- Cooperative Knowledge Distillation: A Learner Agnostic Approach (AAAI 2024 · similarity 0.89)
- PLD: A Choice-Theoretic List-Wise Knowledge Distillation (NeurIPS 2025 · similarity 0.88)
- ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $\alpha$-$\beta$-Divergence (ICML 2025 · similarity 0.88)
- Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation (ICLR 2024 · similarity 0.88)
- Asymmetric Decision-Making in Online Knowledge Distillation: Unifying Consensus and Divergence (ICML 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.