Research map / Large language models / Reasoning and chain of thought

Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

ICML 2026 · Large language models › Reasoning and chain of thought · approach: Test-time compute scaling · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows an exact bijection between autoregressive models and energy-based models via the chain rule of probability (equivalent to the soft Bellman equation), derives equivalence of NLL minima and KL bounds for distilling EBMs into ARMs, and validates on tiny synthetic models.

Problem

Whether autoregressive next-token models can approximate sequence-level energy-based models, and hence can plan ahead despite being trained with next-token prediction; also theoretical justification of teacher forcing and of distilling EBMs into ARMs in MaxEnt RL post-training.

Approach

Using the chain rule of probability, define an explicit bijection M between a per-token reward r and an ARM logit q (q = r + soft value of the next state), showing that the ARM's logits implicitly encode the EBM's log-partition as a lookahead value function; this recovers the soft Bellman equation in the tabular, finite-horizon DAG setting.

Key results

• Proves an exact bijection M between EBM per-token rewards and ARM logits, with the EBM log-partition equal to the ARM soft value V_q(x).
• Shows minimum NLL of ARMs and EBMs coincide in function space, implying teacher forcing is optimal under this setting.
• Bounds KL(ARM, EBM) by 2T times the max L-infinity logit error; numerical experiments on tiny models show ARM and EBM loss curves nearly coincide.

Surprising findings

• The EBM and ARM loss curves are very close throughout training, not just at convergence.

Experimental setup

Datasets: synthetic data

Metrics: optimality gap of expected NLL L-infinity distance between logits KL divergence bound

Builds on: maximum entropy RL soft Bellman equation KL-regularized RL

Closest papers by meaning

• Beyond Autoregression: Fast LLMs via Self-Distillation Through Time (ICLR 2025 · similarity 0.88)
• ShiQ: Bringing back Bellman to LLMs (NeurIPS 2025 · similarity 0.87)
• Autoregressive Learning in Joint KL: Sharp Oracle Bounds and Lower Bounds (NeurIPS 2026 · similarity 0.86)
• The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning (NeurIPS 2025 · similarity 0.86)
• On Predictability of Reinforcement Learning Dynamics for Large Language Models (ICLR 2026 · similarity 0.86)
• BOND: Aligning LLMs with Best-of-N Distillation (ICLR 2025 · similarity 0.86)