Research map / Large language models / Reasoning and chain of thought

Making Expert Reasoning Learnable with Self-Distillation

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes DAIL, an offline self-distillation method that expands expert solutions into in-distribution reasoning traces via a teacher conditioned on the solution, then trains with a contrastive KL objective that penalizes imitating shortcut-laden negative-reference outputs.

Problem

Expert human solutions are out-of-distribution and didactic for LLMs, so naive imitation (SFT) degrades reasoning, and RLVR gets zero signal on problems too hard for the model to sample correct answers; expert data is also expensive and scarce.

Approach

Use the frozen student as a teacher conditioned on the expert solution, generated via mixed policy rollouts (student tokens accepted when teacher probability exceeds a threshold), to create in-distribution traces; then train with a contrastive token-level KL objective that pulls the student toward the full-information teacher and pushes it away from a negative reference conditioned only on answer

Key results

• DAIL on Qwen2.5-7B-Instruct with e1-verifiable reaches 63.3 pass@128 on AIME 2024/2025 versus 51.7 for the base model, and 46.0 vs 41.0 on BeyondAIME at pass@128.
• On Qwen3 models trained on e1-proof, DAIL achieves roughly the base model's coverage with about 2x fewer reasoning tokens.
• GPQA-Diamond pass@128 for Qwen3 improves to 96.0-96.9 with DAIL versus 93.4-95.5 for the base across token budgets.

Surprising findings

• Direct SFT on expert human solutions and STaR rationalization both degrade performance relative to the untrained model.
• DAIL underperforms the base model at pass@1 on several math benchmarks (e.g., IMO-AnswerBench 6.4 vs 6.8) while improving pass@128.

Limitations

• DAIL fails to improve weaker models such as Llama-3.1-8B-Instruct on BeyondAIME, showing dependence on base model strength.
• Results rely on a small number of tuned hyperparameters (gamma, tau) selected on validation sets and mainly on math benchmarks.
• Expert solutions for e1-verifiable are community-generated AoPS solutions rather than official ones.

Experimental setup

Datasets: e1-verifiable (417 AIME problems 1985-2023 with AoPS solutions) e1-proof (669 olympiad proofs from Evan Chen's site) Reasoning Gym Maze and Rotten Oranges HealthBench consensus subset Tool Alpaca

Benchmarks: AIME 2024/2025 BeyondAIME IMO-AnswerBench GPQA-Diamond Reasoning Gym Maze Reasoning Gym Rotten Oranges HealthBench Hard Tool Alpaca

Metrics: pass@k pass@1 token-budget coverage

Baselines: Qwen2.5-7B-Instruct base direct SFT on expert solutions STaR rationalization GRPO NuRL + GRPO GRPO (DeepScaleR) Qwen3 distillation from Qwen3-235B-A22B

Builds on: GRPO NuRL STaR speculative decoding LoRA

Closest papers by meaning

• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models (ICML 2026 · similarity 0.89)
• Hypothesis, Verification, and Induction: Grounding Large Language Models with Self-Driven Skill Learning (AAAI 2024 · similarity 0.89)
• Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion (ICLR 2026 · similarity 0.88)
• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.88)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.88)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.88)