Research map / Large language models / Reasoning and chain of thought

It Helps to Take a Second Opinion: Teaching Smaller LLMs To Deliberate Mutually via Selective Rationale Optimisation

ICLR 2025 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes COALITION, which trains two variants of the same small LM on separate data splits and uses cross-refinement of rationales, DPO on likelihood-based utility scores, and a deberta controller to pick variants, improving SLM rationale quality without external teachers.

Problem

Smaller language models struggle to explore and self-refine diverse reasoning rationales, and existing approaches rely on larger teacher LLMs, ground-truth rationales, or LLM-as-a-judge, which fail at small scale and face licensing constraints.

Approach

Create two distinct variants of the same SLM by instruction-tuning on disjoint data splits, let them generate and cross-refine rationales, score candidates by the likelihood of the ground-truth answer given the rationale, optimise with DPO (Selective Rationale Optimisation), and route inference through a trained controller that picks which variant generates and which refines.

Key results

• COALITION on Llama3-8B reaches 81.06 on GSM8K and 83.26 on PIQA, improving over the best baseline SPIN (77.01 and 79.02) by about 4 points.
• Across Phi3-3.8B, Qwen1.5 (4B/7B/14B), and Mistral-7B, COALITION improves accuracy on all five reported tasks, e.g. Qwen1.5-14B GSM8K 69.37 to 74.88.
• Ablation shows cross-refinement between variants beats single-model self-refinement (e.g. GSM8K 79.94 vs 77.35), and the controller-routed setting gives the best result (81.06).

Measured effects reported in the paper

Surprising findings

• LLM-as-a-judge rating of rationales by the same small model degrades performance relative to likelihood-based utility, indicating prompt-based rating fails at SLM scale.
• Self-Rewarding LMs and Exchange-of-Thought underperform even the plain Llama3-8B-Instruct without rationales on several tasks.

Limitations

• Evaluated mainly on five multiple-choice or short-answer reasoning datasets; generalisation to open-ended generation is not shown.
• Relies on ground-truth answers to compute likelihood-based utility scores, so requires labelled end-task data.
• Training cost is multiplied by two variants, two DPO iterations and a controller, and the paper reports 8 A100 GPUs.

Experimental setup

Datasets: CoT-Collection Dolly-HHRLHF Open Assistant GSM8K PIQA WinoGrande CSQA HellaSwag

Benchmarks: GSM8K WinoGrande PIQA HellaSwag CSQA MMLU ARC TruthfulQA

Metrics: accuracy

Baselines: Chain-of-Thought CoT Self-Consistency Tree-of-Thought Exchange-of-Thought Self-Refine Distilling Step-by-Step Self-Rewarding LMs SPIN

Builds on: DPO Chain-of-Thought Self-Refine SPIN Distilling Step-by-Step

Closest papers by meaning

• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.90)
• Distilling LLM Agent into Small Models with Retrieval and Code Tools (NeurIPS 2025 · similarity 0.88)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate (ICML 2024 · similarity 0.88)
• STEER: Assessing the Economic Rationality of Large Language Models (ICML 2024 · similarity 0.88)
• Select to Think: Unlocking SLM Potential with Local Sufficiency (ICML 2026 · similarity 0.88)
• MiniPLM: Knowledge Distillation for Pre-training Language Models (ICLR 2025 · similarity 0.88)