Research map / Large language models / Reasoning and chain of thought

Energy-Based Transformers are Scalable Learners and Thinkers

ICLR 2026 · Large language models › Reasoning and chain of thought · approach: Efficient and adaptive thinking

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Introduces Energy-Based Transformers (EBTs), a Transformer-based energy-based model trained purely unsupervised to scale faster than Transformer++ and DiT, and to gain System 2 Thinking via energy minimization at inference.

Problem

Existing System 2 Thinking methods are modality-specific, problem-specific, or need extra supervision (verifiers or verifiable rewards); EBMs have historically struggled to scale, so it is unclear whether unsupervised learning alone can yield System 2 Thinking across modalities.

Approach

Train Transformers as energy-based models that assign a scalar energy to each input and candidate-prediction pair; predictions are made by gradient-descent energy minimization (with the loss backpropagated through the optimization), so the same model is both generator and verifier, and extra computation at inference means more optimization steps or more candidates ranked by energy.

Key results

• EBTs scale up to 35% faster than Transformer++ across data, batch size, parameters, FLOPs, and depth during pretraining.
• System 2 Thinking (more optimization steps) improves EBT language perplexity by up to 29% while Transformer++ cannot improve; self-verification gains reach 10-14% at larger data scale.
• On image denoising EBTs reach PSNR 27.25 vs DiT 26.58 in distribution and 23.29 vs 19.56 OOD using 99% fewer forward passes.

Surprising findings

• Transformer++ does not improve at all with extra per-token forward passes, whereas EBTs do.
• EBTs achieve lower downstream perplexity on most tasks (GSM8K, BB Math, BB Dyck) despite a slightly higher pretraining perplexity than Transformer++.

Limitations

• Hyperparameters such as step size and optimization steps add tuning burden and affect training stability; training is more expensive due to gradients of gradients.
• Largest model is 800M parameters, far below foundation-model scale; scaling beyond 10^21 FLOPs untested.
• Struggles with multimodal distributions (e.g., class-conditional or text-to-image generation), producing blurry outputs.

Experimental setup

Datasets: RedPajamaV2 FineWeb Something Something V2 COCO 2014 ImageNet-1k GSM8K SQuAD BigBench Elementary Math QA BigBench Dyck Languages

Benchmarks: GSM8K SQuAD BigBench Elementary Math QA BigBench Dyck Languages COCO 2014 denoising ImageNet-1k linear probe Something Something V2

Metrics: perplexity validation loss PSNR MSE top-1 accuracy top-5 accuracy scaling rate

Baselines: Transformer++ Diffusion Transformer (DiT)

Builds on: Transformer++ Diffusion Transformer Langevin dynamics GPT BERT

Closest papers by meaning

• Transformers Learn Faster with Semantic Focus (NeurIPS 2025 · similarity 0.86)
• The Hidden Joules: Evaluating the Energy Consumption of Vision Backbones for Progress Towards More Efficient Model Inference (ICML 2025 · similarity 0.85)
• Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces (ICLR 2025 · similarity 0.85)
• Only Strict Saddles in the Energy Landscape of Predictive Coding Networks? (NeurIPS 2024 · similarity 0.85)
• Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models (NeurIPS 2025 · similarity 0.85)
• VFScale: Intrinsic Reasoning through Verifier-Free Test-time Scalable Diffusion Model (ICLR 2026 · similarity 0.85)