Research map / Large language models / Reasoning and chain of thought

Markovian Transformers for Informative Language Modeling

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Long CoT dynamics and efficiency · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes a Markovian LM framework where the answer is predicted from a bounded-length CoT alone, trained with a GRPO-style policy gradient including actor-reward gradients, showing gains on QA tasks and higher CoT-perturbation sensitivity than non-Markovian baselines.

Problem

Chain-of-thought explanations often fail to be causally load-bearing: models can ignore or bypass the CoT because they still see the original question when producing the answer.

Approach

An autoencoder-style text bottleneck: the answer predictor sees only the CoT (not the question), and the CoT generator is trained with GRPO-style parallel sampling, a frozen-baseline CoT', within-batch standardized advantages, and direct gradients through the actor reward (chain rule), plus a KL penalty to the pretrained model.

Key results

• Markovian training raises GSM8K accuracy from 19.6% to 57.1% and ARC-Challenge from 36.1% to 79.9% for Llama 3.1 8B, versus 63.3% and 78.6% for the Non-Markovian variant on GSM8K and ARC.
• Mean accuracy across five QA datasets is 66.6% for Markovian vs 70.2% for Non-Markovian (Llama 3.1 8B).
• Wikipedia perturbation analysis shows Markovian checkpoints have larger log-probability drops than Non-Markovian across all perturbation types and severities (column means up to +0.926 for delete).

Surprising findings

• On Arithmetic, the Non-Markovian model is marginally more sensitive to CoT perturbation than the Markovian model (-1.5 pp for Markovian), and the paper attributes this to conflicts with the intact question.
• On MMLU, the Markovian model (55.5%) trails the Non-Markovian model (68.7%) by about 13 pp, larger than the average gap.

Limitations

• Constraint ensures CoT sufficiency, not faithfulness to the model's internal computation; post-hoc CoTs remain possible.
• Interpretability is verified only via perturbation fragility and cross-model transfer; no human studies.
• Markovian training trails the Non-Markovian variant on MMLU and average accuracy.

Experimental setup

Datasets: GSM8K MMLU SVAMP ARC-Challenge Wikipedia synthetic arithmetic

Benchmarks: GSM8K MMLU SVAMP ARC-Challenge Arithmetic Wikipedia continuation

Metrics: accuracy log-probability drop under CoT perturbation normalized log-likelihood (nats)

Baselines: Base model (untrained) Expert Iteration Non-Markovian GRPO No Reward Gradient ablation EMA baseline Policy Gradient

Builds on: GRPO chain-of-thought prompting LoRA expert iteration

Closest papers by meaning

• Eliciting Chain-of-Thought in Base LLMs via Gradient-Based Representation Optimization (AAAI 2026 · similarity 0.92)
• SCOUT: Teaching Pre-trained Language Models to Enhance Reasoning via Flow Chain-of-Thought (NeurIPS 2025 · similarity 0.91)
• Towards Efficient Large Language Reasoning Models via Extreme-Ratio Chain-of-Thought Compression (ICML 2026 · similarity 0.91)
• Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis (ICLR 2025 · similarity 0.91)
• SIM-CoT: Supervised Implicit Chain-of-Thought (ICLR 2026 · similarity 0.90)
• Metastable Dynamics of Chain-of-Thought Reasoning: Provable Benefits of Search, RL and Distillation (ICML 2025 · similarity 0.90)