Research map / Architectures and efficiency / Attention and transformers
Truth as a Compression Artifact in Language Model Training
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains small transformers from scratch on contradictory math and Wikipedia corpora and finds models prefer correct answers when errors are random but not when errors form one coherent wrong rule, with two competing wrong rules restoring the preference.
Problem
When transformers train on contradictory data (same problem with correct and incorrect solutions), which answer they prefer and why; specifically whether truth preference depends on the structure/compressibility of errors.
Approach
Frames next-token prediction as compression/MDL and hypothesizes models prefer the answer system with lower description length; varies error structure (random vs single coherent rule vs multiple rules) at equal frequency to show truth bias vanishes for coherent errors and returns with multi-rule errors.
Key results
- Random errors yield paired accuracy rising from 65.3% (3.5M) to 85.2% (86M), while single coherent rule stays near chance (43.5%-51.0%).
- Multi-rule errors: accuracy jumps from 46.6% with one rule to 77.6% with two rules and 88.3% with ten.
- At ~1B scale on FineWeb-Edu with 8% contradictory math, random errors give 76.8% vs coherent 46.7% (single seed per condition).
Surprising findings
- Coherent errors yield below-chance accuracy at tiny scale (~43%), and with a coherent-error majority the model follows frequency (72% preference for false system at 40/60).
- Verification-based recovery of truth bias shows inverse scaling, dropping from 71% at 3.5M to 61% at 86M.
Limitations
- Largest primary models are 86M parameters; the ~1B experiment uses a single seed and only 8% contradictory math.
- Random and coherent corruptions differ in other ways (step count, surface simplicity), so causal identification is partial.
- Paired accuracy is far higher than generative accuracy (85% vs 53% at large), limiting claims about output truthfulness.
Experimental setup
Datasets: SymPy-generated math problems Wikipedia paragraphs (entity substitution) FineWeb-Edu
Benchmarks: synthetic math paired evaluation Wikipedia entity-substitution paired evaluation FineWeb-Edu + math paired evaluation
Metrics: pair accuracy generative accuracy (SymPy-verified) delta loss gzip compression ratio
Baselines: random-error corpus coherent-error corpus
Builds on: minimum description length Qwen3 FineWeb-Edu
Closest papers by meaning
- Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs (ICLR 2026 · similarity 0.88)
- Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only (ICLR 2025 · similarity 0.87)
- Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Models (ICLR 2024 · similarity 0.87)
- Distinguishing the Knowable from the Unknowable with Language Models (ICML 2024 · similarity 0.87)
- Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimension (ICML 2024 · similarity 0.87)
- Alignment for Honesty (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.