Research map / Large language models / Reasoning and chain of thought

Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Pretrains Llama-style LMs from scratch on synthetic and FB15K-237 knowledge-graph triples to study the smallest model size that achieves best implicit multi-hop reasoning, and links that optimal size linearly to a graph search entropy measure (~0.008 bits per parameter).

Problem

How much model capacity is needed for implicit reasoning (inferring unseen multi-hop facts from learned knowledge without CoT) to emerge during pretraining, and what determines the minimal sufficient model size?

Approach

Shows a U-shaped test-loss vs model-size curve under prolonged training, defines an optimal (smallest sufficient) model size, and proposes graph search entropy (from maximal-entropy random walks on the KG) as the complexity measure that scales linearly with that optimal size, with a theoretical argument and a fitted scaling law.

Key results

• Optimal model size correlates linearly with graph search entropy with R^2=0.85 on synthetic graphs, and the FB15K-237 point falls close to the fitted line.
• Scaling fit implies about 124 extra parameters per 1-bit entropy increase, i.e. reasoning capacity of roughly 0.008 bits per parameter, far below the 2 bits/parameter knowledge capacity reported elsewhere.
• Test loss shows a U-shaped curve vs model size under long training; larger models match but do not beat the best loss and may overfit.

Surprising findings

• Under long training, larger models do not outperform the optimal-size model, and the best achievable reasoning loss is set by data rather than model size.
• Random-ID tokenization gave cleaner scaling trends than natural-language triple verbalization.

Limitations

• Results are on simplified knowledge-graph pretraining data and may not transfer to real text pretraining.
• Optimal size is discretized over selected model sizes and measured at a finite 10k-step budget, adding noise to the fit.
• Theoretical upper and lower bounds rely on strong assumptions (finite-precision capacity, sparse basis approximation) and the paper itself notes the bounds are loose and architecture-dependent.

Experimental setup

Datasets: FB15K-237 FB15K (source of FB15K-237) synthetic knowledge graphs

Benchmarks: synthetic KG implicit reasoning test set (10-option multiple choice, M=1000) FB15K-237 held-out triples

Metrics: multiple-choice accuracy next-token test loss graph search entropy vs optimal model size R^2

Builds on: Llama FB15K-237 Chinchilla scaling laws Physics of language models (knowledge graph reasoning)

Closest papers by meaning

• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.91)
• Reasoning with Latent Thoughts: On the Power of Looped Transformers (ICLR 2025 · similarity 0.91)
• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.90)
• T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling (ICML 2025 · similarity 0.90)
• Let Me Think! A Long Chain of Thought Can Be Worth Exponentially Many Short Ones (NeurIPS 2025 · similarity 0.90)
• Scaling Reasoning Hop Exposes Weaknesses: Demystifying and Improving Hop Generalization in Large Language Models (ICLR 2026 · similarity 0.90)