Research map / Large language models / Reasoning and chain of thought

Select to Think: Unlocking SLM Potential with Local Sufficiency

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Select to Think (S2T), where an LLM ranks an SLM's top-K next-token candidates at divergence points, and S2T-Local distills this selection into the SLM via reserved-token scores so it runs without LLM calls.

Problem

Small language models lag behind LLMs on multi-step reasoning; existing collaborative methods need costly LLM generation at divergence points, and standard distillation struggles with the capacity gap over the open-ended vocabulary.

Approach

Observes "local sufficiency": at divergence points the LLM's preferred token is usually within the SLM's top-K candidates. Reframes LLM help as selection among SLM proposals (discrete ranking) rather than generation, and distills that ranking into the SLM by repurposing reserved vocabulary logits as preference scores, decoupling selection from generation.

Key results

• S2T-Local improves the 1.5B SLM's Math Avg. from 36.9 to 45.8 (24.1% relative) over greedy decoding, matching Maj@8 (43.7) at single-trajectory cost.
• The 1.5B SLM's top-8 candidates contain the 32B teacher's choice in about 95% of divergence steps, while Hit@1 is about 30%.
• The distilled selector reaches Agree@1 of 69.4% on MATH500, 71.4% on OlympiadBench and 67.2% on MMLU-Pro, versus 12.5% random.

Surprising findings

• Local sufficiency holds even for the 0.5B SLM, with Hit@8 above 83%, despite its much smaller size.
• Top-p=0.7 sampling for candidate generation truncates the candidate set to about 1.5 items on average and removes teacher-preferred tokens, hurting selection.

Limitations

• Relies on a 32B teacher for annotation, and trigger/divergence calibration uses KL divergence to the teacher.
• Head-only tuning and matching teacher scores were found insufficient, requiring LoRA with discrete selection loss and margin filtering.
• Some results are weaker on AIME25 and OlympiadBench, and S2T-Local trails R2R and SpecReason on several benchmarks.

Experimental setup

Datasets: MATH training split

Benchmarks: GSM8K MATH500 OlympiadBench AIME25 HumanEval MMLU-Pro

Metrics: accuracy Math Avg. pass@1 Hit@K Agree@1 Spearman rho tokens per instance wall-clock time

Baselines: Greedy decoding Sample DistilQwen2.5 Maj@8 self-consistency TSD-KD Multiplex-Thinking Think-at-Hard R2R SpecReason LLM-Takeover

Builds on: Zero-overhead Inference-time Prediction (ZIP) TSD-KD Self-Consistency SpecReason Qwen2.5

Closest papers by meaning

• Scalable Best-of-N Selection for Large Language Models via Self-Certainty (NeurIPS 2025 · similarity 0.89)
• It Helps to Take a Second Opinion: Teaching Smaller LLMs To Deliberate Mutually via Selective Rationale Optimisation (ICLR 2025 · similarity 0.88)
• LensLLM: Unveiling Fine-Tuning Dynamics for LLM Selection (ICML 2025 · similarity 0.88)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.87)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.87)
• MILES: Modular Instruction Memory with Learnable Selection for Self-Improving LLM Reasoning (NeurIPS 2026 · similarity 0.87)