AI Paper Atlas

Research map / Large language models / Language, safety and interpretability

Residual Energy-Based Models for Text

Anton Bakhtin, Yuntian Deng, Sam Gross, Myle Ott, Marc'Aurelio Ranzato, Arthur Szlam

JMLR 2021 · Large language models › Language, safety and interpretability · approach: Controlled text generation · 22 citations (OpenAlex)

Read the paper

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
replace causal unidirectional energy network with bidirectional BiT-Base (ablation)residual EBM · CC-News test perplexity (12L)perplexity: 15.57-15.58 -> 14.61-14.64Improved
add residual EBM (joint UniT) on top of base LMBase LM · CC-News test perplexity (12L)perplexity: 17.57 -> 15.57-15.58Improved
residual EBM (joint UniT) versus equal-size BALMjoint UniT · Toronto Book Corpus test perplexity (12L)perplexity: 16.99 vs 16.98-17.00No clear effect
make base LM deeper (24 layers instead of 12) (ablation)BALM · CC-News test perplexityperplexity: 15.74 -> 13.92Improved
residual EBM on 24-layer base LM, joint UniT versus BALM-24Ljoint UniT · Toronto Book Corpus test perplexityperplexity: 18.24 -> 17.46-17.48Improved
use RALM residual autoregressive model instead of base LM (24L)Base LM-24L · CC-News test perplexityperplexity: 14.89 -> 14.89No clear effect
initialize BiT-Base from RoBERTa (external data) (ablation)Joint BiT-Base · Toronto Book Corpus test perplexity (12L)perplexity: 15.72-15.74 vs no external data not reportedImproved
use RoBERTa-Large (BiT-Large*) instead of RoBERTa-Base (BiT-Base*), 24L base (ablation)Joint BiT · CC-News test perplexityperplexity: 12.93-12.95 -> 12.10-12.16Improved
use external-data pretrained BiT-Base* instead of BiT-Base with 24L base LM (ablation)Joint BiT-Base · CC-News test perplexityperplexity: 13.01-13.03 -> 12.93-12.95Improved
human evaluation: joint BiT-Base versus base LMjoint BiT-Base · CC-News human preferencepreference rate: 56.25%Improved
human evaluation: joint BiT-Large* versus base LMjoint BiT-Large* · CC-News human preferencepreference rate: 58.93%Improved
human evaluation: joint UniT versus base LMjoint UniT · CC-News human preferencepreference rate: 52.85%No clear effect

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy