Research map / Optimization / Implicit bias and generalization

On the Origins of Linear Representations in Large Language Models

ICML 2024 · Optimization › Implicit bias and generalization · approach: Explicit and implicit regularization interplay · 2 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces a latent binary concept variable model of next-token prediction and shows that log-odds matching and the implicit bias of gradient descent (with softmax cross-entropy) both promote linear and orthogonal concept representations, with simulations and LLaMA-2 checks.

Problem

Why are high-level semantic concepts encoded linearly in LLM representation spaces, and could apparent linearity be illusory? The paper seeks theoretical origins of linear representations under the next-token prediction objective.

Approach

Model context and next tokens as sharing a latent space of binary concept variables forming a Markov random field, then show that softmax cross-entropy with gradient descent's implicit bias (max-margin convergence of exponential loss) yields parallel steering vectors and aligned embedding/unembedding representations, with orthogonality for separated concepts.

Key results

• In simulations trained on the complete set of conditionals, average cosine similarities among unembedding steering vectors range from 0.972 to 0.997 for m=3 to 7 variables.
• LLaMA-2-7B shows alignment between embedding and unembedding steering vectors for matching concepts relative to unmatched concepts on multilingual translation pairs.
• Theory shows that under gradient descent with exponential loss, cosine similarity between same-concept steering vectors converges to 1 (max-margin direction).

Measured effects reported in the paper

Surprising findings

• The orthogonal initialization assumption in the gradient-descent alignment theorem appears unnecessary according to the authors' experiments.
• Final cosine similarities in LLaMA-2 experiments are lower than in simulations, which the authors attribute to natural language complexity and abstract concepts not being one-dimensional.

Limitations

• Assumes injectivity from concepts to tokens and a simplified binary latent concept structure, which may not hold for LLMs.
• Log-odds and fixed-embedding analyses are simplified subproblems; the full nonconvex training dynamics are not analyzed.
• LLM experiments yield only partial alignment, with lower cosine similarities than simulations and abstract concepts may not be one-dimensional.

Experimental setup

Datasets: OPUS Books Big Analogy Test dataset Winograd Schema dataset synthetic samples from random DAG latent variable models

Benchmarks: OPUS Books Big Analogy Test dataset Winograd Schema dataset

Metrics: average cosine similarity between steering vectors absolute cosine similarity between embedding and unembedding

Builds on: word2vec linear structure analyses RAND-WALK model max-margin implicit bias of gradient descent

Closest papers by meaning

• The Geometry of Categorical and Hierarchical Concepts in Large Language Models (ICLR 2025 · similarity 0.90)
• The Linear Representation Hypothesis and the Geometry of Large Language Models (ICML 2024 · similarity 0.88)
• Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models (NeurIPS 2025 · similarity 0.88)
• On Linear Representations and Pretraining Data Frequency in Language Models (ICLR 2025 · similarity 0.88)
• Emergence of Linear Truth Encodings in Language Models (NeurIPS 2025 · similarity 0.87)
• The Foundations of Tokenization: Statistical and Computational Concerns (ICLR 2025 · similarity 0.87)