Research map / Large language models / Language, safety and interpretability

Tracing the Representation Geometry of Language Models from Pretraining to Post-training

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Linguistic encoding in LLM representations · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Tracks RankMe and αReQ spectral geometry of last-token representations across OLMo-2, Pythia pretraining and Tülu-3.1 post-training, finding three pretraining phases (warmup, entropy-seeking, compression-seeking) and that SFT/DPO expand while RLVR compresses representations.

Problem

Training loss decreases monotonically and fails to explain when LLMs acquire complex capabilities; it is unclear how representation geometry evolves during pretraining and post-training and how it relates to capabilities.

Approach

Uses spectral metrics of the last-token representation covariance (effective rank RankMe and power-law eigenspectrum decay αReQ) to reveal a non-monotonic three-phase geometry during pretraining, links the phases to n-gram memorization and downstream QA gains, and provides a linear cross-entropy toy model explaining the phases under skewed token frequencies and a low-dimensional bottleneck.

Key results

• On OLMo-2 7B and Pythia, RankMe and αReQ vary non-monotonically across pretraining while loss decreases monotonically, with warmup, entropy-seeking and compression-seeking phases.
• Retaining only top-10 eigen-directions drops Pythia-1B SciQ accuracy from 0.838 to 0.225 and OLMo-2-7B from 0.970 to 0.155.
• In Tülu-3.1-8B post-training, RankMe increases under SFT and DPO and decreases under RLVR; RLVR-trained model underperforms base model on AMC-23 pass@256.

Measured effects reported in the paper

Surprising findings

• RLVR-trained model is outperformed by the base and intermediate checkpoints at pass@256 on AMC-23 despite winning at pass@16.
• Removing top eigen-directions has minimal impact on SciQ accuracy, suggesting information is distributed across the full spectrum.

Limitations

• Models limited to up to 12B parameters; spectral metric computation requires ~10K samples and scales quadratically with hidden dimension.
• Theory assumes linear feature extractors and i.i.d. samples, not full transformers.
• Findings are correlational rather than causal; English-only models with standard objectives.

Experimental setup

Datasets: FineWeb sample-10BT Anthropic-HH AlpacaFarm Tulu3 mixture TriviaQA SciQ WikiText LAMBADA OpenAI AMC23 AlpacaEval

Benchmarks: SciQ TriviaQA AlpacaEval AMC23

Metrics: RankMe αReQ Spearman correlation (memorization) accuracy pass@k win rate ID/OOD loss

Baselines: loss-based training metrics

Builds on: RankMe αReQ infinity-gram DPO Tülu 3

Closest papers by meaning

• Less is More: Local Intrinsic Dimensions of Contextual Language Models (NeurIPS 2025 · similarity 0.89)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations (NeurIPS 2026 · similarity 0.88)
• A Phase Transition between Positional and Semantic Learning in a Solvable Model of Dot-Product Attention (NeurIPS 2024 · similarity 0.88)
• When Less is More: The LLM Scaling Paradox in Context Compression (NeurIPS 2026 · similarity 0.88)
• Geometry of Decision Making in Language Models (NeurIPS 2025 · similarity 0.87)
• Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.87)