Research map / Large language models / Reasoning and chain of thought

Latent Concept Disentanglement in Transformer-based Language Models

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Search and self-correction in reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Uses activation patching and probing to show that Gemma-2 models resolve latent bridge concepts in 2-hop ICL and that small transformers encode numerical latent parameters (add-k, circular/rectangular trajectories) as low-dimensional, geometry-preserving task vectors.

Problem

How transformers represent and compose latent concepts inferred from in-context demonstrations, especially when a hidden bridge entity (2-hop world knowledge) or a continuous numerical parameter must be inferred for ICL.

Approach

Applies causal mediation (interchange activation patching) to locate sparse attention heads that resolve a latent bridge concept in 2-hop ICL, and shows that task vectors for small models trained from scratch lie on low-dimensional manifolds that mirror the latent parameter geometry, with steering that interpolates the parameter.

Key results

• On Gemma-2-27B, head group (24,30;31) causally transfers the bridge concept across source/target types; patching boosts the type-corrected alternative answer into top-10 at least 73% of the time and to top-1 more than 40% of the time on [University, Code]->[City, Capital].
• On add-k tasks, the first principal component of task vectors explains more than 99.9% of variance, with offset ordering preserved along the line.
• For circular trajectories, the first two PCs explain 97.05%, 96.44%, and 93.68% of variance across K=16, 32, 64 radii, and steering with interpolated task vectors moves the output radius toward the target.

Surprising findings

• Gemma-2-27B's bridge-resolving heads transfer the bridge concept even when source and target types have no overlap, rather than the model just emitting the literal alternative answer.
• Scaling the patched head representation by a constant around 2.0 often improves intervention results.

Limitations

• Analysis of the 2-layer from-scratch models is restricted to simple synthetic numerical tasks.
• Smaller Gemma-2-2B results are weaker and noisier, so the size-dependence conclusion is a conjecture.
• Evaluation of intervention uses a non-standard type-corrected metric rather than literal alternative answers.

Experimental setup

Datasets: Geography 2-hop ICL puzzles (City/University/Landmark to Capital/Calling Code) Company 2-hop ICL puzzles synthetic add-k sequences synthetic circular and rectangular trajectory sequences

Benchmarks: Geography 2-hop ICL puzzles Company 2-hop ICL puzzles add-k Circular-Trajectory Rectangular-Trajectory

Metrics: logit difference reciprocal rank top-1 accuracy top-3 accuracy MSE of steered radius PCA variance explained linear probe accuracy

Baselines: Gemma-2-2B

Builds on: activation patching function vectors task vectors TwoHopFact

Closest papers by meaning

• Why Struggle with Continuous Latents? Interpretable Discrete Latent Reasoning via Rendered Compression (NeurIPS 2026 · similarity 0.89)
• Learning to Disentangle Latent Reasoning Rules with Language VAEs: A Systematic Study (AAAI 2026 · similarity 0.89)
• Provably Transformers Harness Multi-Concept Word Semantics for Efficient In-Context Learning (NeurIPS 2024 · similarity 0.88)
• Think in Latent, Explain in Language: Self-Explainable Latent Reasoning (ICML 2026 · similarity 0.88)
• Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers (NeurIPS 2024 · similarity 0.88)
• Geometric Latent Reasoning Induces Shorter Generations in LLMs (NeurIPS 2026 · similarity 0.88)