Research map / Large language models / Reasoning and chain of thought

AlgoTrace: Algorithmic Primitives and Compositional Geometry of Reasoning in Language Models

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Clusters LLM latent activations during multi-step reasoning into algorithmic primitives, extracts function-vector-style primitive vectors, and tests their injection, steering, cross-task transfer, and arithmetic composition in Phi-4 family, Llama-3-8B, and Qwen models.

Problem

How LLMs implement multi-step reasoning via latent algorithmic primitives, whether these primitives are geometrically composable and transferable across tasks, and how reasoning finetuning changes primitive usage and composition.

Approach

Combine k-means clustering of residual-stream activations over reasoning traces, GPT-4o-based labeling of clusters, spectral meta-clustering of cluster transitions, and function-vector-style primitive vectors from top ICL attention heads that can be injected, subtracted, and summed to steer reasoning behavior across tasks.

Key results

• Injecting the nearest_neighbor primitive vector raised the proportion of nearest-neighbor paths by 56.1% and compare_verify raised comparison words by 1103.7% over baseline in Phi-4 on TSP.
• Summing a TNR and an RC primitive vector in zero-shot Llama-3-8B nearly matches few-shot performance on the composite task.
• Phi-4-Reasoning responses on TSP were highly stereotyped, while Phi-4 responses split into brute-force and random-guess groups.

Surprising findings

• Subtracting a compute_distance primitive vector reduced distance computations, suggesting negative steering prevents primitives.
• Compositional induction by vector addition worked for some models (Llama-3-8B) but not others, suggesting models implement composition differently.

Limitations

• Primitives do not always map to known algorithms and were identified via manual and GPT-4o-based labeling.
• Only linear compositions of primitive vectors were studied; nonlinear interactions left for future work.
• Small evaluation scale (around 5 responses for clustering, 100 per task) and a single layer (17) for clustering.

Experimental setup

Datasets: TSP prompts (self-generated, 100+ examples per task) 3-SAT problems AIME 2025 problems GraphNav binary tree problems in-context TNR and RC tasks

Benchmarks: TSP 3-SAT AIME GraphNav

Metrics: proportion of nearest-neighbor paths number of unique valid paths token position of distance computation final answer token position verification and comparison word counts chi-squared dissimilarity of cluster frequencies cluster expressivity

Baselines: Phi-4 Phi-4-Reasoning

Builds on: function vectors in-context task vectors steering vectors

Closest papers by meaning

• Geometric Latent Reasoning Induces Shorter Generations in LLMs (NeurIPS 2026 · similarity 0.89)
• Boule or Baguette? A Study on Task Topology, Length Generalization, and the Benefit of Reasoning Trace (NeurIPS 2026 · similarity 0.89)
• Compositional Causal Reasoning Evaluation in Language Models (ICML 2025 · similarity 0.88)
• Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problems (ICLR 2026 · similarity 0.88)
• Compositional Reasoning in Language Models under Reinforcement Learning Post-Training (NeurIPS 2026 · similarity 0.88)
• Benchmarking Abstract and Reasoning Abilities Through A Theoretical Perspective (ICML 2025 · similarity 0.88)