Research map / Large language models / Language, safety and interpretability

It's All Training: A Fully Synthetic Single-Stage Recipe for LLMs

NeurIPS 2026 · Large language models › Language, safety and interpretability · approach: LLM unlearning and benchmarks · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Synth, an open fully synthetic single-stage pretraining corpus back-translated from 58k Wikipedia seeds, and the Baguettotron model suite (56M to 13B MoE) trained on it without SFT or RL.

Problem

Web-crawled pretraining data lacks explicit reasoning and post-training-style behaviors, and the effect of synthetic data on knowledge and skill acquisition in small LMs is poorly understood; frontier-lab synthetic datasets are not public.

Approach

Back-translate instruction, RAG, reasoning, arithmetic, creative writing, editing and MCQ tasks from curated Wikipedia seed passages using constraint-prior sampling and a stenographic reasoning syntax, so a single pretraining stage yields instruction-following, reasoning, and grounded factual recall without SFT or RL.

Key results

• Baguettotron-600M reaches 41.7% macro FActScore and Baguettotron-MoE 46.3%, above Phi-4-mini-instruct (29.5%) and DeepSeek-MoE-16B-Chat (38.3%) on Wikipedia entities.
• In a data ablation at 600M, Synth without post-training scores 42.2% on 22 MCQ tasks versus 26.1% (FineWiki) and 25.1% (FinePDFs-Edu) with SmolTalk post-training.
• Baguettotron-600M with tool-calling data reaches 53.1% on BFCL v2, 3.9 points above FunctionGemma-270M, on ~40x fewer training tokens.

Surprising findings

• Removing reasoning traces leaves multiple-choice accuracy essentially unchanged (41.8% vs 42.2%) but lowers open-ended accuracy by 2.3 points, mainly on TruthfulQA and NuclearQA.
• On held-out entities Baguettotron-MoE abstains on 67% of queries, while the web-trained OLMoE abstains on only 7%.

Limitations

• Knowledge ceiling bounded by the ~58k Wikipedia seed articles, limiting broad web-knowledge benchmarks such as ARC-Challenge and GeoBench.
• Seeding is shaped by English Wikipedia contributors, so cultural diversity is limited.
• Factual precision evaluated with an LLM judge and FActScore-style protocol; Baguettotron is far behind web-trained models on coverage of held-out entities.

Experimental setup

Datasets: Synth Wikipedia vital articles Wikibooks Common Corpus 3GPP technical standards SmolTalk MMLU auxiliary train split FineWiki FinePDFs-Edu BFCL v2

Benchmarks: MMLU ARC-Easy ARC-Challenge TruthfulQA NuclearQA TeleQnA BFCL v2 NQ-Open PopQA TriviaQA WikiFact SimpleQA

Metrics: accuracy FActScore precision S/(S+C) macro FActScore abstention rate micro accuracy macro accuracy

Baselines: Gemma-3-270M SmolLM2-360M LFM2.5-350M Qwen3-0.6B OLMoE-1B-7B-Instruct DeepSeek-MoE-16B-Chat Phi-4-mini-instruct FineWiki FinePDFs-Edu FunctionGemma-270M OPT-350M

Builds on: Phi-1.5 Cosmopedia Chinchilla scaling laws FActScore EntiGraph

Closest papers by meaning

• Unveiling the Secret Recipe: A Guide For Supervised Fine-Tuning Small LLMs (ICLR 2025 · similarity 0.91)
• Fixing It in Post: A Comparative Study of LLM Post-Training Data Quality and Model Performance (NeurIPS 2025 · similarity 0.90)
• Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (ICLR 2025 · similarity 0.90)
• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective (ICLR 2025 · similarity 0.90)
• Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning (NeurIPS 2025 · similarity 0.90)
• Exploring the Limits of Sub-Billion Language Model Reasoners with Open Training Recipes (ICLR 2026 · similarity 0.89)