Research map / Large language models / Reasoning and chain of thought

Syntax vs. Semantics: How Transformers Learn Deep Dependencies

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Theory of autoregressive reasoning · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a spectral/gradient-starvation framework for why transformers learn deep dependencies late (phase transition), uses it to motivate CoT and a topology-aligned contrastive attention loss, tested on toy transformers, Pythia checkpoints, and Llama-3.1-8B / Qwen2.5-Coder-7B variable cloze.

Problem

Why transformers learn deep semantic dependencies (e.g., variable binding) only after a delay and abrupt transition while syntactic/surface patterns are learned early, and why CoT helps.

Approach

Models training as competition between high-curvature surface-syntax gradients and low-curvature semantic gradients (gradient starvation), with softmax saturation as a gate; motivates a contrastive margin loss on top-8 binding heads that steers attention toward definition tokens.

Key results

• On Variable Cloze Completion with Llama-3.1-8B, the proposed contrastive objective reaches 0.362 on Python vs 0.335 base and 0.343 SFT, with improvement over base more than twice SFT's gain.
• On Qwen2.5-Coder-7B Python, Ours scores 0.909 vs 0.890 base and 0.893 SFT.
• On WinoGrande, Ours improves Llama-3.1-8B from 0.710 base (0.863 SFT) to 0.896 and Qwen2.5-7B from 0.679 base (0.845 SFT) to 0.874.

Surprising findings

• Masking the top-8 binding heads causes a larger drop than random masking, but Top8 ablation results are reported only in a single table without variance.
• The semantic-operator effective rank of ~1.19 vs ~62 for syntactic heads is reported as evidence of rank collapse.

Limitations

• Theoretical propositions rely on local quadratic approximations and assumptions (isotropy, orthogonality) that are only loosely verified empirically.
• LLM experiments are limited to code and a single coreference-based natural language extension; gains are small in absolute terms (e.g., +0.027 on Llama Python).
• Heads selected via heuristic Def/Use labeling (first occurrence = Def) and no variance or multiple seeds are reported for main results.

Experimental setup

Datasets: Python150k JavaScript150k WinoGrande The Pile (Pythia pretraining) AllenNLP coreference extraction

Benchmarks: Variable Cloze Completion (Python150k, JavaScript150k) WinoGrande

Metrics: accuracy

Baselines: SFT Curriculum learning Linear-chain CRF SupCon Attention entropy regularization Top8 head ablation Random head ablation

Builds on: Gradient Starvation Chain-of-Thought prompting LoRA Pythia

Closest papers by meaning

• From Sparse Dependence to Sparse Attention: Unveiling How Chain-of-Thought Enhances Transformer Sample Efficiency (ICLR 2025 · similarity 0.88)
• Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.87)
• A Lens into Interpretable Transformer Mistakes via Semantic Dependency (ICML 2025 · similarity 0.87)
• Attention Illuminates LLM Reasoning: The Uncovered Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization (ICML 2026 · similarity 0.87)
• Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization (NeurIPS 2025 · similarity 0.87)
• When More is Less: Understanding Chain-of-Thought Length in LLMs (ICLR 2026 · similarity 0.87)