Research map / Large language models / Reasoning and chain of thought

Reusing Pre-Training Data at Test Time is a Compute Multiplier

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that retrieving from the same open pre-training corpora used to train models (plus test-time self-consistency and reranking) yields large MMLU, Math-500, SimpleQA and GPQA gains, framing retrieval as a ~5x compute multiplier over pre-training.

Problem

Pre-training may not extract all the knowledge and value contained in pre-training datasets; the paper asks how much value is left behind and how it scales, by reusing pre-training data via retrieval and test-time compute.

Approach

Pre-train dense models at several compute budgets on open corpora, then retrieve from the identical corpora at test time (embedding + reranker, n-gram decontamination), quantify retrieval as a compute multiplier by fitting sigmoid curves to base MMLU vs FLOPs, and stack self-consistency, MMR and bagging on top of retrieval with Llama 3.1 8B as reader.

Key results

• Retrieval from the pre-training corpus acts as an average ~4.86x compute multiplier over pre-training on MMLU (sigmoid fit), declining to 2.88x at the largest scale.
• With Llama 3.1 8B, retrieval plus reranker reaches 74.0% on SimpleQA, and reranker plus self-consistency plus variance reduction reaches 82.1% on MMLU (vs 71.6% baseline).
• Math-500 rises from 48.7% baseline to 64.4% with reranker, self-consistency and variance reduction; GPQA all rises from 30.6% to 36.8%.

Surprising findings

• Better pre-training data (DCLM, 53.4 pre-training MMLU) is not necessarily better for retrieval; FineWeb-edu (42.9 pre-training MMLU) matches or slightly beats it as a retrieval store.
• Retrieval gains are larger for STEM than for humanities, despite retrieval being expected to help mainly with factual recall.

Limitations

• Only simple test-time techniques and a limited set of evaluations are explored.
• Math-500 and MMLU show n-gram contamination (32.0% and 14.1% of test items found in pre-training corpora), so decontamination only partly addresses leakage.
• Compute multiplier estimates rely on a sigmoid fit to a limited set of five model scales, and the Llama 3.1 8B estimate is an extrapolation given its overtraining.

Experimental setup

Datasets: DCLM-baseline FineWeb-edu arXiv peS2o PubMed Central Stack Exchange Wikipedia AlgebraicStack AutoMathText FineMath OpenWebMath StackMathQA

Benchmarks: MMLU Math-500 SimpleQA GPQA LiveCodeBench

Metrics: accuracy compute multiplier

Baselines: Llama 3.1 8B Instruct (no retrieval) Qwen3 Reranker 0.6B GPT-4.1 mini (checker) gpt-4o-2024-08-06 (LiveCodeBench)

Builds on: self-consistency retrieval-augmented generation Honeycrisp (Axlearn)

Closest papers by meaning

• Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scaling (ICLR 2026 · similarity 0.88)
• Inference Scaling for Long-Context Retrieval Augmented Generation (ICLR 2025 · similarity 0.88)
• Bridging Compute- and Data-Optimal Pretraining (NeurIPS 2026 · similarity 0.88)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (ICLR 2025 · similarity 0.87)
• Training with “Paraphrasing the Original Text” Teaches LLM to Better Retrieve in Long-Context Tasks (AAAI 2025 · similarity 0.87)
• InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining (ICML 2024 · similarity 0.87)