Research map / Large language models / Language, safety and interpretability

Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers

NeurIPS 2025 · Large language models › Language, safety and interpretability · approach: Reasoning mechanisms and failure modes · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows that out-of-context reasoning drives both generalization and hallucination in LLM knowledge injection, and proves via implicit bias that a factorized one-layer attention model learns OCR while the combined-weight model cannot.

Problem

Why fine-tuning on new facts makes LLMs both generalize to implications and hallucinate incorrect implications, and what mechanism explains this duality.

Approach

Treats generalization and hallucination as one mechanism, out-of-context reasoning (OCR). Shows theoretically that gradient descent on a factorized output-value parameterization implicitly minimizes the nuclear norm of the combined matrix, enabling OCR, whereas the non-factorized matrix minimizes the Frobenius norm and cannot generalize.

Key results

• In LLMs, models rank implications for test subjects well for causally related pairs (e.g., Gemma-2-9B generalization mean rank 0.00) and also show hallucination on fictitious unrelated pairs (e.g., Qwen-2-7B mean rank 4.55 on one hallucination pair, Mistral 2.10).
• Factorized one-layer attention model reaches zero test loss while the non-factorized model fails to generalize (training with |S|=80, n=20, m=4, d=128).
• Theory: non-factorized model's test loss is at least log|A_2| under trainable key-query, and the factorized SVM gives margin lower bound min{sqrt(m_train/m_test),1} on test implications versus zero for the non-factorized model.

Measured effects reported in the paper

Surprising findings

• Reparameterization W_OV = W_O W_V^T, usually considered expressivity-equivalent, changes optimization and generalization outcomes: the non-factorized model cannot do OCR.
• Hallucination arises even for causally unrelated fact-implication pairs, since any nonzero training subject ratio yields positive margin.

Limitations

• Theory is for one-layer attention-only linear models with fixed or partially trainable attention; multi-layer transformers left to future work.
• Key-query training analysis shows only the non-factorized case; factorized model with trainable W_KQ not fully analyzed.
• Gradient-flow convergence direction for factorized multi-class case is conjectured rather than proven.

Experimental setup

Datasets: synthetic fictitious-name fact/implication dataset (City-Language, Country-Code, Profession-Color, Sport-Music, counterf symbolic factual recall sequences

Benchmarks: synthetic OCR factual recall task

Metrics: mean rank of implication train/test cross-entropy loss margin

Baselines: non-factorized one-layer attention model (W_OV)

Builds on: reversal curse factual recall transformers (Nichani et al.) implicit bias of gradient descent on matrix factorization SVM equivalence for homogeneous networks

Closest papers by meaning

• Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization (NeurIPS 2024 · similarity 0.89)
• Mitigating Hallucinations in Large Language Models via Causal Reasoning (AAAI 2026 · similarity 0.88)
• Extrapolation by Association: Length Generalization Transfer In Transformers (NeurIPS 2025 · similarity 0.88)
• Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers (NeurIPS 2024 · similarity 0.88)
• Understanding the Role of Hallucination in Reinforcement Post-Training of Multimodal Reasoning Models (CVPR 2026 · similarity 0.87)
• Connecting the Dots: LLMs can Infer and Verbalize Latent Structure from Disparate Training Data (NeurIPS 2024 · similarity 0.87)