Research map / Large language models / Alignment and preferences

Thinking LLMs: General Instruction Following with Thought Generation

ICML 2025 · Large language models › Alignment and preferences · approach: Active and token-level preference learning · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Thought Preference Optimization (TPO), which prompts an instruction-tuned LLM to think before responding and iteratively optimizes the thoughts with DPO using a judge that scores only the final responses, yielding gains on AlpacaEval and Arena-Hard for general instruction following.

Problem

Instruction-tuned LLMs respond directly without explicit thinking, and there is no labeled thought data for general instruction following, so it is unclear how to train models to think before answering beyond math and logic tasks.

Approach

Bootstrap thoughts via thought prompts on a seed model, then optimize thoughts indirectly by judging only the response part (hiding thoughts from the judge) and applying iterative DPO on thought+response pairs, with length control to prevent verbosity.

Key results

• TPO reaches 52.5% AlpacaEval LC win rate vs 48.4% for the direct response baseline (Llama-3-8B-Instruct seed at 24.9%).
• TPO reaches 37.3% Arena-Hard win rate vs 33.0% for the direct baseline.
• Thinking hurts at iteration 0 (17.3% vs 24.9% AlpacaEval) but surpasses the direct baseline after 3-4 iterations; GSM8K accuracy degrades under the setup (e.g., 70.6% vs 79.2% original prompts).

Measured effects reported in the paper

Surprising findings

• Thinking gains appear in non-reasoning categories such as marketing, health, language and general knowledge, not only math and reasoning.
• Simply prompting the seed model to think lowers AlpacaEval win rate from 24.9% to 17.3%, and the thought part shrinks during training (61% generic, 30% specific) without direct supervision.

Limitations

• Math performance degrades on GSM8K, attributed to few math training instructions (2.2%) and lack of a math-capable judge.
• Thought length is not controllable by the user; longer thoughts add compute cost.
• All experiments use 8B models; effect on larger models is untested.

Experimental setup

Datasets: UltraFeedback Open Assistant-based synthetic instructions (Llama-2-70B-Chat generated) GSM8K

Benchmarks: AlpacaEval 2 Arena-Hard GSM8K UltraFeedback fine-grained 20-category evaluation

Metrics: length-controlled win rate win rate accuracy ArmoRM score

Baselines: Llama-3-8B-Instruct Direct response baseline GPT-4 (06/13) Llama-3-70b-instruct Mistral Large (24/02) Qwen2 72B Instruct

Builds on: DPO Chain-of-Thought STaR IRPO Self-Taught Evaluator

Closest papers by meaning

• Thinkless: LLM Learns When to Think (NeurIPS 2025 · similarity 0.88)
• Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.87)
• Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.87)
• Think-J: Learning to Think for Generative LLM-as-a-Judge (AAAI 2026 · similarity 0.87)
• StrategyLLM: Large Language Models as Strategy Generators, Executors, Optimizers, and Evaluators for Problem Solving (NeurIPS 2024 · similarity 0.87)
• J1: Incentivizing Thinking in LLM-as-a-Judge via Reinforcement Learning (ICLR 2026 · similarity 0.87)