Research map / Large language models / Language, safety and interpretability
Residual Energy-Based Models for Text
Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc'Aurelio Ranzato, Arthur Szlam
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace causal unidirectional energy network with bidirectional BiT-Base (ablation) | residual EBM · CC-News test perplexity (12L) | perplexity: 15.57-15.58 -> 14.61-14.64 | Improved |
| add residual EBM (joint UniT) on top of base LM | Base LM · CC-News test perplexity (12L) | perplexity: 17.57 -> 15.57-15.58 | Improved |
| residual EBM (joint UniT) versus equal-size BALM | joint UniT · Toronto Book Corpus test perplexity (12L) | perplexity: 16.99 vs 16.98-17.00 | No clear effect |
| make base LM deeper (24 layers instead of 12) (ablation) | BALM · CC-News test perplexity | perplexity: 15.74 -> 13.92 | Improved |
| residual EBM on 24-layer base LM, joint UniT versus BALM-24L | joint UniT · Toronto Book Corpus test perplexity | perplexity: 18.24 -> 17.46-17.48 | Improved |
| use RALM residual autoregressive model instead of base LM (24L) | Base LM-24L · CC-News test perplexity | perplexity: 14.89 -> 14.89 | No clear effect |
| initialize BiT-Base from RoBERTa (external data) (ablation) | Joint BiT-Base · Toronto Book Corpus test perplexity (12L) | perplexity: 15.72-15.74 vs no external data not reported | Improved |
| use RoBERTa-Large (BiT-Large*) instead of RoBERTa-Base (BiT-Base*), 24L base (ablation) | Joint BiT · CC-News test perplexity | perplexity: 12.93-12.95 -> 12.10-12.16 | Improved |
| use external-data pretrained BiT-Base* instead of BiT-Base with 24L base LM (ablation) | Joint BiT-Base · CC-News test perplexity | perplexity: 13.01-13.03 -> 12.93-12.95 | Improved |
| human evaluation: joint BiT-Base versus base LM | joint BiT-Base · CC-News human preference | preference rate: 56.25% | Improved |
| human evaluation: joint BiT-Large* versus base LM | joint BiT-Large* · CC-News human preference | preference rate: 58.93% | Improved |
| human evaluation: joint UniT versus base LM | joint UniT · CC-News human preference | preference rate: 52.85% | No clear effect |
Closest papers by meaning
- Energy-Based Diffusion Language Models for Text Generation (ICLR 2025 · similarity 0.89)
- Principled Gradient-Based MCMC for Conditional Sampling of Text (ICML 2024 · similarity 0.86)
- Language Model Decoding as Direct Metrics Optimization (ICLR 2024 · similarity 0.86)
- Scaling Diffusion Language Models via Adaptation from Autoregressive Models (ICLR 2025 · similarity 0.86)
- Reflection-Window Decoding: Text Generation with Selective Refinement (ICML 2025 · similarity 0.86)
- Idiosyncrasies in Large Language Models (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
