Research map / Architectures and efficiency / Attention and transformers
NRGPT: An Energy-based Alternative for GPT
Nima Dehmamy, Benjamin Hoover, Bishwajit Saha, Leo Kozachkov, Jean-Jacques Slotine, Dmitry Krotov
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use NRGPT energy-based block instead of recurrent GPT block | recurrent GPT (GPT_Rec_parallel) · OpenWebText n_embed=768 | validation loss: 3.454 -> 3.467 | No clear effect |
| use NRGPT instead of 12-layer GPT | GPT (124M) · OpenWebText n_embed=768 | validation loss: 2.921 -> 3.467 | Hurt |
| use NRGPT instead of GPT for generation | GPT · OpenWebText generation | distinct-1: 0.619 -> 0.674 (distinct-1); perplexity 75 -> 104 | Mixed |
| use NRGPT instead of GPT_Rec_parallel on ListOps | recurrent GPT · ListOps | parameters at learning transition: 2.3 -> 2.4 (FF1) / 2.98 (FF2W), x10^4 | No clear effect |
| use NRGPT instead of GPT on Shakespeare at large sizes | GPT · Shakespeare | overfitting / validation loss | Improved |
| widen FF1 hidden dimension from 4D to 8D (ablation) | NRGPT_H_FF1 · ListOps | competitiveness with Rec-GPT | Improved |
Closest papers by meaning
- Energy-Based Transformers are Scalable Learners and Thinkers (ICLR 2026 · similarity 0.88)
- LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Models (ICLR 2024 · similarity 0.86)
- Entropy-Gated Latent Recursion (NeurIPS 2026 · similarity 0.86)
- ConsistentEE: A Consistent and Hardness-Guided Early Exiting Method for Accelerating Language Models Inference (AAAI 2024 · similarity 0.86)
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.86)
- Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
