Research map / Large language models / Reasoning and chain of thought

GSM-$\infty$: How Do your LLMs Behave over Infinitely Increasing Reasoning Complexity and Context Length?

ICML 2025 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces GSM-∞, a synthetic long-context math reasoning benchmark generated from computational graphs with controllable reasoning complexity (operation count) and injected noise, and evaluates 18 LLMs on zero-noise and 8K–32K settings.

Problem

Existing long-context benchmarks have low reasoning complexity, detectable noise, and limited quantity of hard examples, so they cannot quantitatively measure how LLM reasoning degrades with increasing complexity and context length.

Approach

Generate grade-school math problems from randomly sampled computational graphs, with implicit +, −, ×, ÷ operations via abstract parameters and a reverse mode, and inject noise as "spider topology" extensions of the core graph that RAG retrievers cannot separate from essential content, giving unlimited difficulty and context length.

Key results

• DeepSeek-R1 achieves a zero-context average AUC of 9427-scale score of 0.9427 versus 0.2186 for Llama3.1-8B-Instruct and 0.6924 for Gemini-1.5-Pro-002 in the Table 1 leaderboard (Score column).
• Gemini-1.5-Pro-002 is the top model in the long-context table, with average AUC 963.9 across 8K, 16K, and 32K, ahead of Qwen2.5-72B-Instruct (724.17).
• Repeated sampling gives AUC that is linear in log(N) with R-squared above 0.99, meaning exponentially increasing compute yields linear AUC gains.

Measured effects reported in the paper

Surprising findings

• Reverse-direction problems are harder than forward ones for most LLMs, with a sigmoid curve left-shifted relative to forward problems.
• RAG systems match or beat long-context LLMs on RULER, LongBench, and LOFT, yet fail to separate GSM-∞ noise from essential content.

Limitations

• Only grade-school arithmetic operations (+, −, ×, ÷) are covered, not richer math operators.
• Synthetic templates limit natural-language diversity; noise variables sometimes produce confusing irregular values.
• Smaller models fail by 32K and larger models were too costly to evaluate at long contexts, so long-context coverage is limited.

Experimental setup

Datasets: GSM-8K

Benchmarks: GSM-∞ (Symbolic, Medium, Hard) RULER LongBench LongBench v2 LOFT

Metrics: area under curve (AUC) over operation count accuracy sigmoid fit R^2 pass@k-style repeated sampling accuracy

Baselines: RULER LongBench LongBench v2 LOFT OnePassRAG InteractiveRAG

Builds on: GSM-8K Ye et al. 2024 abstract/instance parameter framework RULER FLARE active RAG best-of-N repeated sampling

Closest papers by meaning

• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (ICLR 2025 · similarity 0.95)
• ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoning (ICML 2025 · similarity 0.92)
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic (NeurIPS 2024 · similarity 0.91)
• Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• AbstRaL: Augmenting LLMs' Reasoning by Reinforcing Abstract Thinking (ICLR 2026 · similarity 0.89)
• Putnam-AXIOM: A Functional & Static Benchmark for Measuring Higher Level Mathematical Reasoning in LLMs (ICML 2025 · similarity 0.89)