Research map / Large language models / Instruction and fine-tuning
Think Thrice Before You Act: Progressive Thought Refinement in Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Progressive Thought Refinement (PTR), building a multi-round refinement dataset via weak-strong model collaboration and fine-tuning Qwen2-7B and Llama3-8B with thought-masked weighted loss to improve answers over iterations across ten tasks.
Problem
LLMs' progressive self-refinement typically depends on supervision signals or task-specific pipelines, which do not generalize to open-ended tasks or new domains.
Approach
Build a annotation-free progressive refinement dataset (thoughts from a weak model, refined answers from a strong model, filtered by thought-answer consistency) and fine-tune with a thought-mask weighted loss so the model learns to improve prior thoughts rather than to predict correct answers.
Key results
- PTR raises the ten-task average for Qwen2-7B from 49.6% (iteration 1) to 53.5% (iteration 3), with MMLU rising from 57.1% to 64.1%.
- For Llama3-8B, PTR average improves from 55.8% to 58.6% over iterations, while the Prompt baseline drops from 55.2% to 47.1% at iteration 2.
- IFT and RL (DPO) baselines degrade across iterations on both models, e.g., RL on Qwen2-7B falls from 48.9% to 42.9%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train with PTR and refine over three iterations rather than use the base-model prompt | Qwen2-7B · MMLU | accuracy: 57.1 -> 64.1 | Helped |
| Train with PTR and refine over three iterations rather than use the base-model prompt | Qwen2-7B · Overall average across ten tasks | average score: 49.6% to 53.5% | Helped |
| Continue refinement from Iter. 1 to Iter. 3 (ablation) | PTR on Qwen2-7B · MMLU | accuracy: 59.2 -> 63.2 | Helped |
| Continue refinement from Iter. 1 to Iter. 3 (ablation) | PTR on Qwen2-7B · HumanEval | Pass@1: 52.3 -> 57.6 | Helped |
| Continue refinement from Iter. 1 to Iter. 3 (ablation) | PTR on Qwen2-7B · Winogrande | accuracy: 66.4 -> 66.8 | No clear effect |
| Use later refinement iterations rather than the initial answer (ablation) | PTR on Qwen2-8B · GSM8K | accuracy: 76.7 -> 79.9 | Helped |
| Use later refinement iterations rather than the initial answer (ablation) | PTR on Qwen2-8B · CommonsenseQA | accuracy: 47.9 -> 58.6 | Helped |
| Use Iteration 3 rather than Iteration 2 (ablation) | PTR on Qwen2-8B · Overall average | average score: 53.65 to 53.33 | Hurt |
| Use temperature 0 rather than higher or decaying temperature (ablation) | PTR inference · Three-iteration evaluation | accuracy | Helped |
| Vary the refinement instruction among Prompts 1, 2, and 3 (ablation) | PTR · GSM8K | accuracy: 75.1; 77.9; 77.5 | Mixed |
Surprising findings
- Plain prompting for refinement degrades performance on both base models (e.g., Llama3-8B MMLU falls from 66.4% to 34.4% at iteration 2).
- Iteration gains plateau after about three rounds, and the ten-iteration study shows marginal changes thereafter.
Limitations
- Wilcoxon significance test and the strong-vs-weak model comparison are described inconsistently; the text says the test was 'expected' rather than reported.
- Evaluation relies on GPT-4 and Auto-j judgments for open-ended quality, with limited details on the judging protocol.
Experimental setup
Datasets: WizardLM (Evol-Instruct V2 196k) MMLU HumanEval DROP XSum GSM8K MATH ARC GPQA Winogrande CommonsenseQA
Benchmarks: MMLU HumanEval DROP XSum GSM8K MATH ARC GPQA Winogrande CommonsenseQA
Metrics: accuracy pass@1 similarity (BGE-m3)
Baselines: Prompt IFT RL (DPO)
Builds on: Evol-Instruct (WizardLM) DPO in-context learning
Closest papers by meaning
- Rethinking Thinking Tokens: LLMs as Improvement Operators (ICML 2026 · similarity 0.90)
- PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space (ICML 2026 · similarity 0.90)
- Thinker: Learning to Think Fast and Slow (NeurIPS 2025 · similarity 0.89)
- Latent Thinking Optimization: Your Latent Reasoning Language Model Secretly Encodes Reward Signals in its Latent Thoughts (ICLR 2026 · similarity 0.88)
- Supervised Knowledge Makes Large Language Models Better In-context Learners (ICLR 2024 · similarity 0.88)
- Latent Thought Models with Variational Bayes Inference-Time Computation (ICML 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.