Research map / Large language models / Reasoning and chain of thought
Keypoint-based Progressive Chain-of-Thought Distillation for LLMs
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes KPOD, a chain-of-thought distillation framework that weights rationale tokens via learned mask learning and schedules an easy-to-hard in-rationale progressive distillation using a submodular question-selection value function.
Problem
Standard chain-of-thought distillation treats all rationale tokens equally and trains students on all steps at once, ignoring keypoint token significance and the easy-to-hard order of reasoning steps.
Approach
A rationale token weighting module learns Gumbel-softmax masks of reasoning-irrelevant tokens (answer prediction loss plus mask ratio loss) whose keep-probabilities weight the distillation loss; the student is then trained progressively, first on the last steps given preceding steps, extending to the full rationale, with question selection governed by a monotone submodular value function solved by
Key results
- On LLaMA-7B, KPOD reaches 46.74% on GSM8K, versus 41.58% for the second-best MCC-KD.
- On FlanT5-XL, KPOD reaches 88.04% on CommonsenseQA, versus 83.62% for SCOTT.
- OOD transfer trained on GSM8K with LLaMA-7B: KPOD gets 57.96 on ASDiv and 47.33 on SVAMP, the best among baselines.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add token significance weighting during distillation (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 42.64 -> 46.74 | Helped |
| Add token significance weighting when calculating step difficulty (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 44.01 -> 46.74 | Helped |
| Add in-rationale progressive distillation (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 43.25 -> 46.74 | Helped |
| Add diversity term to question-set selection (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 44.16 -> 46.74 | Helped |
| Replace Adaptive CL with in-rationale progressive distillation (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 43.55 -> 46.74 | Helped |
| Replace SPL with in-rationale progressive distillation (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 42.94 -> 46.74 | Helped |
| Replace ICL with in-rationale progressive distillation (ablation) | KPOD with LLaMA-7B · GSM8K | accuracy (%): 43.85 -> 46.74 | Helped |
| Add token significance weighting during distillation (ablation) | KPOD with FlanT5-XL · CommonsenseQA | accuracy (%): 85.26 -> 88.04 | Helped |
| Add token significance weighting when calculating step difficulty (ablation) | KPOD with FlanT5-XL · GSM8K | accuracy (%): 23.82 -> 25.19 | Helped |
| Add in-rationale progressive distillation (ablation) | KPOD with FlanT5-XL · CommonsenseQA | accuracy (%): 84.28 -> 88.04 | Helped |
| Add diversity term to question-set selection (ablation) | KPOD with FlanT5-XL · GSM8K | accuracy (%): 23.98 -> 25.19 | Helped |
| Replace Adaptive CL with in-rationale progressive distillation (ablation) | KPOD with FlanT5-XL · GSM8K | accuracy (%): 23.52 -> 25.19 | Helped |
Limitations
- Relies on a single teacher (GPT-3.5-Turbo) and zero-shot CoT rationales.
- Results are reported on small benchmarks with relatively small validation/test splits, and gains vary by dataset.
- Hyperparameter sensitivity was studied only on GSM8K with LLaMA-7B.
Experimental setup
Datasets: GSM8K ASDiv SVAMP CommonsenseQA
Benchmarks: GSM8K ASDiv SVAMP CommonsenseQA
Metrics: accuracy
Baselines: SCoTD Specialized KD SCOTT MCC-KD Adaptive CL SPL ICL GPT-3.5-Turbo
Builds on: Specialized KD SCOTT MCC-KD SCoTD zero-shot chain-of-thought FTGP
Closest papers by meaning
- SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.90)
- Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy (ICLR 2026 · similarity 0.90)
- Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.89)
- Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.89)
- Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.89)
- KaVa: Latent Reasoning via Compressed KV-Cache Distillation (ICLR 2026 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.