Research map / Large language models / Reasoning and chain of thought

Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Builds MobileLLM-R1, a family of sub-billion-parameter reasoning models pretrained on ~4.2T tokens of curated open data using influence-based data mixing and iterative mid-training data compression.

Problem

Whether strong reasoning in small (sub-billion-parameter) language models requires massive pretraining corpora (>10T tokens), given that such models are sensitive to data noise and must be deployable on-device.

Approach

Uses leave-one-out NLL analysis on capability-probing datasets and AutoMixer-style influence scores (self- and cross-capability) to set dataset-level mixing weights without benchmark data, then iteratively compresses mid-training data by keeping only positive-influence samples while co-evolving with the model.

Key results

• MobileLLM-R1-950M achieves an AIME'24 score of 15.5 versus 0.6 for OLMo-2-1.48B and 0.3 for SmolLM2-1.7B.
• MobileLLM-R1-950M reaches 46.3% HumanEval (base) versus 30.5% for Qwen3-0.6B-Base, using 4.2T tokens vs 36T.
• MobileLLM-R1-950M scores 74.0 on MATH500 and 19.9 on LCBv6, surpassing Qwen3-0.6B's 73.0 and 14.9.

Surprising findings

• Adding math or code reasoning SFT data reduces MMLU performance, especially in smaller models.
• StarCoder improves math more than OpenWebMath improves code, contrary to the common assumption.

Limitations

• Relies on reasoning SFT datasets and Tulu-3 for post-training; gains partly depend on post-training data quality.
• Math-focused SFT trades off MMLU factual knowledge in small models.
• Many results are reported on partially overlapping or heuristically chosen capability-probing sets, and the mixing method requires training separate domain checkpoints.

Experimental setup

Datasets: StarCoder OpenWebMath FineWeb-Edu Wikipedia Arxiv StackExchange Algebraic Stack FineMath Nemotron-CC-Math Nemotron-Code Cosmopedia Natural Reasoning

Benchmarks: MATH500 GSM8K AIME'24 AIME'25 LiveCodeBench v6 HumanEval MBPP MMLU CommonSense reasoning (ARC, BoolQ, PIQA, SIQA, HellaSwag, OBQA, WinoGrande)

Metrics: pass@1 exact match accuracy perplexity average accuracy

Baselines: Qwen3-0.6B OLMo-2-1.48B SmolLM2-1.7B SmolLM2-135M SmolLM2-360M Gemma-3-270M LLaMA3.2-1B DeepSeek-R1-Distill-Qwen-1.5B Qwen3-1.7B

Builds on: AutoMixer OLMo 2 MobileLLM Ask-LLM FineWeb-Edu classifier RankMe

Closest papers by meaning

• Training Language Models to Reason Efficiently (NeurIPS 2025 · similarity 0.94)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens (ICML 2026 · similarity 0.93)
• Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning (ICML 2025 · similarity 0.93)
• Long Chain-of-Thought Reasoning Across Languages (ICLR 2026 · similarity 0.92)
• metabench - A Sparse Benchmark of Reasoning and Knowledge in Large Language Models (ICLR 2025 · similarity 0.92)
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought (ICLR 2026 · similarity 0.91)