Research map / Large language models / Reasoning and chain of thought

The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 1 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that LLM long-horizon failures stem from execution errors (not reasoning), isolating execution via provided keys and plan on a key-value running-sum task, finding scaling helps, self-conditioning hurts, and thinking mitigates it.

Problem

Whether continued LLM scaling yields diminishing returns for long-horizon tasks, and why LLMs fail on simple tasks made longer; the paper isolates execution (carrying out a given plan with provided knowledge) from planning and reasoning.

Approach

Uses a controlled key-value running-sum task where the plan (keys) and knowledge (dictionary) are given in context, so only execution is tested; shows per-step errors compound into exponentially growing horizon length, identifies a self-conditioning effect where models make more errors when history contains their own errors, and shows thinking removes it.

Key results

• Under the paper's setup, GPT-5 (thinking) executes 2176 steps in a single turn versus 432 for Claude-4 Sonnet, 384 for Grok 4, and 120 for Gemini 2.5 Pro.
• Without CoT, non-thinking frontier models such as DeepSeek-V3 and Kimi K2 fail to execute a turn complexity above six, while DeepSeek-R1 can execute over 100 steps.
• Qwen3 thinking models do not self-condition: turn-100 accuracy stays stable regardless of injected error rate in history, whereas non-thinking models degrade as error rate rises.

Surprising findings

• Scaling model size improves turn-wise execution but does not mitigate self-conditioning, even for 200B+ non-thinking models.
• Majority voting with matched token budget gives only marginal gains, far below CoT or larger models.

Limitations

• Synthetic key-value summation task; improvement may not transfer to real agentic tasks with many possible actions.
• Single-turn task is theoretically parallelizable, so future-proofing is uncertain.
• Accuracy metric does not account for self-correction.

Experimental setup

Benchmarks: Long-Horizon-Execution key-value running-sum task

Metrics: step accuracy turn accuracy task accuracy horizon length

Baselines: DeepSeek-V3 DeepSeek-R1 Kimi K2 Claude-4 Sonnet Grok 4 Gemini 2.5 Pro GPT-5

Builds on: ReAct METR horizon length analysis

Closest papers by meaning

• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.91)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.90)
• Large Language Models Think Too Fast To Explore Effectively (NeurIPS 2025 · similarity 0.90)
• LIFEBENCH: Evaluating Length Instruction Following in Large Language Models (NeurIPS 2025 · similarity 0.90)
• Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning (NeurIPS 2025 · similarity 0.90)
• Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models (ICLR 2025 · similarity 0.90)