Research map / Large language models / Reasoning and chain of thought

LongCoT: Benchmarking Long-Horizon Chain-of-Thought Reasoning

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces LongCoT, a 2,500-problem benchmark across five domains that isolates long-horizon chain-of-thought reasoning, finding frontier models score under 10% on it.

Problem

Existing benchmarks measure long-horizon reasoning only indirectly, via short hard problems or agentic workflows confounded by tools and scaffolds; there is no direct measure of how accurately models reason over very long chains of thought.

Approach

Template-based, expert-designed problems with short prompts and verifiable answers, built as explicit compositional DAGs of individually tractable subproblems or implicit procedural search problems, so that failures isolate long-horizon reasoning rather than local difficulty or tool use. Difficulty scales programmatically with DAG size and search depth.

Key results

• GPT 5.2 achieves 9.83% on LongCoT (62K average output tokens), Gemini 3 Pro 6.08%, Grok 4.1 Fast 2.04%, other models near zero.
• On LongCoT-mini, GPT 5.2 reaches 38.7%, while Kimi K2 and DeepSeek V3.2 reach 7.5% and 8.3%.
• Observed accuracy on composed Omni-Math problems falls well below the independent-error baseline as node count grows from 1 to 35.

Surprising findings

• GPT 5.2 scores 95.7% on isolated subproblems yet accuracy on composed problems degrades far faster than independent error compounding predicts.
• A recursive language model scaffold with code execution improves only on implicit domains (Logic 68.3%, Chess 30.6%) and stays near zero on compositional domains.

Limitations

• Closed-source models hide reasoning traces, limiting qualitative analysis.
• Cost of long generations prevents pass@k and self-consistency experiments.
• Final answers were manually verified, which may not scale.

Experimental setup

Datasets: Omni-Math HLE

Benchmarks: LongCoT LongCoT-mini

Metrics: accuracy output tokens per problem

Baselines: GPT 5.2 Gemini 3 Pro Claude 4.5 Sonnet Grok 4.1 Fast Reasoning DeepSeek V3.2 Kimi K2 Thinking GLM 4.7 RLM

Builds on: Omni-Math Recursive Language Models

Closest papers by meaning

• Unlocking the Capabilities of Thought: A Reasoning Boundary Framework to Quantify and Optimize Chain-of-Thought (NeurIPS 2024 · similarity 0.91)
• Demystifying Long Chain-of-Thought Reasoning (ICML 2025 · similarity 0.91)
• Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.91)
• The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think (ICLR 2026 · similarity 0.90)
• How Far Ahead Do LLMs Plan? Uncovering the Latent Horizon in Chain-of-Thought Reasoning (ICML 2026 · similarity 0.90)
• FaithCoT-Bench: Benchmarking Instance-Level Faithfulness of Chain-of-Thought Reasoning (ICLR 2026 · similarity 0.90)