Research map / Large language models / Reasoning and chain of thought
Enhancing Language Model Reasoning with Structured Multi-Level Modeling
Siheng Xiong, Ali Payani, Faramarz Fekri
Read the paperCheck your idea against 137,217 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a structured multi-level modeling approach to address long-horizon plan failures in long chain-of-thought reasoning of small language models, instead of a single outcome-reward RL policy.
Problem
Long-CoT inference-time scaling with a single outcome-reward RL policy suffers long-horizon plan failures where the implicit plan drifts from any valid strategy, especially severe for smaller LMs.
Approach
Separates reasoning into multiple levels of modeling rather than a single policy trained with outcome-reward RL, so that an explicit plan-level structure guides the long CoT (details not available in the provided abstract).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use Step-DPO with TSMC process preferences instead of DPO in the multi-level model (ablation) | MLR (Llama-3.1-8B) · average of MATH500/AIME24/GPQA/BoardGameQA | Avg. Pass@1: 55.3 -> 66.1 | Improved |
| add high-level planner (multi-level modeling) on top of Step-DPO (ablation) | Llama-3.1-8B · average of 4 benchmarks | Avg. Pass@1: 59.1 -> 66.1 | Improved |
| add high-level planner on top of DPO (ablation) | Llama-3.1-8B · average of 4 benchmarks | Avg. Pass@1: 51.6 -> 55.3 | Improved |
| add iterative Step-DPO after SFT (ablation) | MLR (Llama-3.1-8B) · average of 4 benchmarks | Avg. Pass@1: 42.2 -> 66.1 | Improved |
| add iterative Step-DPO after SFT (ablation) | MLR (Qwen-2.5-1.5B) · average of 4 benchmarks | Avg. Pass@1: 35.8 -> 54.2 | Improved |
| use MLR instead of SFT + GRPO under same data budget | Qwen-2.5-1.5B · average of 4 benchmarks | Avg. Pass@1: 48.4 -> 54.2 | Improved |
| use MLR instead of SFT + GRPO under same data budget | Llama-3.1-8B · average of 4 benchmarks | Avg. Pass@1: 60.0 -> 66.1 | Improved |
| use MLR instead of DeepSeek-R1-Distill with 10% SFT and 5% preference data | Qwen-2.5-MATH-7B · average of 4 benchmarks | Avg. Pass@1: 60.0 -> 66.2 | Improved |
| use Step-DPO instead of DPO on the SFT initialization (ablation) | Llama-3.1-8B (low-level only) · average of 4 benchmarks | Avg. Pass@1: 51.6 -> 59.1 | Improved |
| use Plan-and-Solve prompting | Qwen-2.5-1.5B · AIME24 Pass@1 | Pass@1: 3.3 (Instruct) -> 2.0 | Hurt |
| increase rollout count K in TSMC (ablation) | iterative Step-DPO · preference estimation | estimator variance and compute | Mixed |
| use small 1.5B SFT model as rollout policy instead of larger model (ablation) | TSMC preference construction · preference quality | preference accuracy and runtime | No clear effect |
Closest papers by meaning
- Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.92)
- T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.91)
- Rectifying LLM Thought from Lens of Optimization (ICLR 2026 · similarity 0.91)
- When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.91)
- Markovian Transformers for Informative Language Modeling (ICLR 2026 · similarity 0.91)
- Incentivizing Reasoning for Advanced Instruction-Following of Large Language Models (NeurIPS 2025 · similarity 0.91)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
