Research map / Large language models / Reasoning and chain of thought

Train Once, Answer All: Many Pretraining Experiments for the Cost of One

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Routing, pruning and distillation for reasoning

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes running ten controlled pretraining experiments simultaneously within a single OLMo-2-1B training run (210B tokens), replicating prior findings and testing inter-experiment dependence with a new continual pretraining dependence test (CPDT).

Problem

Controlled pretraining experiments are expensive because each experiment normally requires its own full pretraining run, limiting scientific study of LLM data effects.

Approach

Perform multiple independent pretraining interventions (data insertions, canaries, poisoning, watermarks) within one training run, justified by the multitask nature of pretraining, and test for dependencies before pretraining using continual pretraining dependence testing (CPDT).

Key results

• Ten simultaneous experiments on OLMo-2-1B over 210B tokens replicated 8 prior-work findings, e.g. 74.4% verbatim completion in the verbatim memorization replication.
• Mathematical reasoning experiment raised OLMo-2-1B-Exp accuracy on 1-step problems from 39.2% to 99.6% and on 5-step problems from 4.8%-range baseline to 51.5%.
• Validation loss of OLMo-2-1B-Exp is 2.6100 vs 2.6088 for OLMo-2-1B, and holdout benchmark accuracy is 55.15% vs 55.51%.

Surprising findings

• Language modeling benchmarks show significant dependencies (e.g., training on ARC-Easy raises ARC-Challenge accuracy by 6.2 points), whereas the controlled experiments showed no detectable dependencies.
• Gaussian watermarks gave a stronger privacy signal than the 1-Rare-Token canary in the low-repetition regime, while standard loss-based membership inference failed on non-canary data.

Limitations

• Only one main training run at 1.5B (with 2.7B, 546M, 179M variants); generalization to other architectures and larger scales is not established.
• Verbatim memorization experiment violates the dependence-test assumptions, so CPDT is not appropriate for it.
• Experiments are restricted to language data; transfer to image or tabular domains is untested.

Experimental setup

Datasets: OLMo-mix-1124 DCLM-Baseline fictional knowledge dataset synthetic grade-school math reasoning problems OLMES benchmarks

Benchmarks: ARC-Easy ARC-Challenge BoolQ CSQA HellaSwag OpenBookQA PIQA SocialIQA WinoGrande MMLU NQ DROP

Metrics: accuracy validation loss train loss true positive rate at 1% FPR dependence matrix entries

Baselines: OLMo-2-1B OLMo-2-0425-1B

Builds on: OLMo-2 DCLM-Baseline Gaussian Unlearning Score OLMES

Closest papers by meaning

• How Do Large Language Models Acquire Factual Knowledge During Pretraining? (NeurIPS 2024 · similarity 0.89)
• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.88)
• What Do Learning Dynamics Reveal About Generalization in LLM Mathematical Reasoning? (ICML 2025 · similarity 0.87)
• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory (NeurIPS 2026 · similarity 0.87)
• Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection (ICML 2025 · similarity 0.87)
• Training on the Test Task Confounds Evaluation and Emergence (ICLR 2025 · similarity 0.86)