Research map / Large language models / Reasoning and chain of thought
SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes Flow CoT and SCOUT, a lightweight fine-tuning method that trains recursive reuse of Qwen2.5-0.5B layers with per-iteration distillation from capacity-matched teachers and a cross-attention retrospective module.
Problem
Recursive latent reasoning in LLMs typically needs costly pretraining, applies identical supervision to all iterations, and uses ad hoc ways to pass previous iterations forward; the paper asks how to get depth-aware recursive reasoning via fine-tuning alone.
Approach
Each recursive iteration is supervised by a teacher of matching capacity (progressively larger Qwen2.5 teachers via per-step KL distillation), and a causal cross-attention module lets each iteration attend to the previous iteration's latent state while self-attending over the original input.
Key results
- SCOUT at iteration 3 reaches 39.03 average accuracy vs. 37.21 for SFT, a +1.81 gain (the paper's 'up to 1.8%').
- R-Distill-EQ peaks at +1.23 in iteration 1 then regresses to -0.04 in iteration 2, while SCOUT improves monotonically (+0.23 → +1.05 → +1.81).
- R-SCOUT (reversed teacher order) reaches +1.47 at iteration 1 but drops to -0.42 at iteration 3.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use SCOUT with progressive teacher order (1.5B → 3B → 7B) and cross-attention | Qwen2.5-0.5B · All benchmarks | Avg: 37.21 -> 39.03 | Helped |
| Add recursive fine-tuning with hard-label supervision at each step (ablation) | Qwen2.5-0.5B · All benchmarks | Avg: 37.21 -> 37.49 | Helped |
| Add fixed 7B-teacher distillation with equal iteration weights (ablation) | Recursive Qwen2.5-0.5B · All benchmarks | Avg: 37.21 -> 37.36 | No clear effect |
| Increase loss weights across iterations while retaining a fixed 7B teacher (ablation) | Recursive distillation · All benchmarks | Avg: 37.36 -> 38.08 | Helped |
| Reverse teacher order to 7B → 3B → 1.5B (ablation) | SCOUT · All benchmarks | Avg: 39.03 -> 36.78 | Hurt |
| Add capacity-matched progressive teacher supervision instead of the fixed-teacher control (ablation) | Recursive Qwen2.5-0.5B · All benchmarks | Avg: 38.08 -> 39.03 | Helped |
| Use cross-attention instead of Init, Add, CatProj, Gate, or ModInj integration (ablation) | R-SFT · All benchmarks, iteration 3 | Avg: 27.11 / 29.09 / 23.95 / 27.09 / 28.98 -> 37.50 | Helped |
| Use cross-attention rather than additive fusion (ablation) | R-SFT · MBPP, iteration 1 | pass@1: 28.20 -> 27.40 | Hurt |
| Use cross-attention rather than gate-based control (ablation) | R-SFT · GLUE, iteration 1 | average: 51.08 -> 47.72 | Hurt |
| Increase the head and recursive blocks from one-third to one-half of layers (Case 2) (ablation) | R-SFT with cross-attention · All benchmarks | Avg: 21.52 -> 37.50 | Helped |
| Use Case 2 instead of Case 1 partitioning (ablation) | R-SFT with additive fusion · All benchmarks | Avg: 22.67 -> 29.09 | Helped |
| Use Case 2 instead of Case 1 partitioning (ablation) | R-SFT with Init injection · All benchmarks | Avg: 26.45 -> 27.11 | Helped |
Surprising findings
- Reversing the teacher order (7B→3B→1.5B) gives the best first-iteration result yet falls below SFT by iteration 3.
- The gains are small in absolute terms (about 1-2 points average) and the MMLU/MBPP-style differences are within what looks like noise on small benchmarks.
Limitations
- Fixed number of reasoning iterations (T=3) and manually chosen teachers; no adaptive depth.
- Gains are modest (max +1.8 average) and evaluated on a single 0.5B student with single runs, so significance is unclear.
- Paper notes ablation on layer partition was limited by compute; alternative layer allocations not explored.
Experimental setup
Datasets: Alpaca GPT-4 Alpaca CoT WikiQA CodeAlpaca MathInstruct Dolly
Benchmarks: ARC-easy ARC-challenge OpenBookQA TruthfulQA GSM8K MMLU CoQA GLUE MBPP
Metrics: accuracy F1 pass@1 average accuracy
Baselines: SFT DSFT R-SFT R-Distill-EQ R-Distill-WT R-SCOUT
Builds on: Universal Transformers Looped Transformers knowledge distillation (KL) Adaptive KL (AKL)
Closest papers by meaning
- CoT Vectors: Transferring and Probing the Reasoning Mechanisms of LLMs (ICLR 2026 · similarity 0.93)
- Enhancing Chain of Thought Prompting in Large Language Models via Reasoning Patterns (AAAI 2025 · similarity 0.93)
- Facts Don't Speak Louder than Words: The Behavioral Essence of Long-CoT Distillation (NeurIPS 2026 · similarity 0.93)
- Scaling Code-Assisted Chain-of-Thoughts and Instructions for Model Reasoning (NeurIPS 2025 · similarity 0.92)
- The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.92)
- SIM-CoT: Supervised Implicit Chain-of-Thought (ICLR 2026 · similarity 0.92)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.