Research map / Large language models / Reasoning and chain of thought

Probing to Refine: Reinforcement Distillation of LLM Reasoners via Explanatory Inversion

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Explanatory Inversion (EI) probes and ExGRPO, an RL-based distillation method with a Dialogue Structure Utility bonus, to transfer reasoning from Gemini-1.5-Pro into Gemma-7B and Qwen2.5-7B students.

Problem

Distilled small LLM students memorize superficial patterns and generalize poorly, including failing inverse versions of problems the teacher solves (reversal-curse-like failures).

Approach

Generate teacher-answered "explanatory probes" via 10 template-based transformation rules, then train the student with multi-turn probe dialogues using GRPO plus a bonus rewarding full-dialogue outcomes over partial-dialogue outcomes, with an auxiliary SFT imitation loss.

Key results

• ExGRPO with Gemma-7B-it student averages 67.19% across eight held-in datasets, a +20.39% gain over zero-shot and above RevThink's 61.17%.
• ExGRPO with Qwen2.5-7B-Instruct averages 82.54%, +4.55% over its zero-shot 77.99% and above the best baseline On-Policy Distillation at 81.60%.
• On OOD tasks, ExGRPO on Gemma-7B-it averages 61.76% vs 56.87% for RevThink.

Surprising findings

• RL from a cold start without SFT warm-up causes catastrophic degradation (Qwen avg. -62.00% from zero-shot).
• Vanilla or insufficient SFT does not always improve the student; on GSM8K with Qwen, 1-epoch SFT slightly helps while other settings degrade.

Limitations

• Depends on the quality of the proprietary Gemini-1.5-Pro teacher; probe templates are manually designed.
• OOD evaluation focuses on format-level perturbations rather than broad domain shifts.
• The theorem's assumptions and the reported reward/hyperparameter details (e.g., SFT-aux weight left as placeholder) are not fully specified.

Experimental setup

Datasets: StrategyQA CommonsenseQA ARC-Challenge MATH GSM8K TabMWP ANLI Date Understanding BoolQ OpenBookQA e-SNLI GSM8K-Reversal

Benchmarks: StrategyQA CommonsenseQA ARC-Challenge MATH GSM8K TabMWP ANLI Date Understanding BoolQ OpenBookQA e-SNLI GSM8K-Reversal

Metrics: accuracy

Baselines: Zero-shot SKD Distilling Step-by-Step Rephrase Question Question Augmentation Answer Augmentation RevThink Divide-or-Conquer On-Policy Distillation

Builds on: GRPO RevThink Distilling Step-by-Step Symbolic Knowledge Distillation

Closest papers by meaning

• General-Reasoner: Advancing LLM Reasoning Across All Domains (NeurIPS 2025 · similarity 0.91)
• $\textbf{Re}^{2}$: Unlocking LLM Reasoning via Reinforcement Learning with Re-solving (ICLR 2026 · similarity 0.90)
• Reasoning with Exploration: An Entropy Perspective (AAAI 2026 · similarity 0.90)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning (NeurIPS 2025 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
• Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.90)