Research map / Probabilistic and scientific modeling / Variational and generative sampling
Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Metropolis-Hastings rather than accept every masked-conditional proposal | MLM-based sequence generation · WMT-14 De-En (20), temperature 1.0 | BLEU: 9.03 -> 24.74 | Helped |
| Use Metropolis-Hastings rather than accept every masked-conditional proposal | MLM-based sequence generation · Unconditional generation, temperature 1.0 | GPT-2 sentence perplexity: 314.74 -> 108.12 | Helped |
| Use block proposals with annealed block size (ablation) | Raw Energy MH · WMT-14 De-En (20) | BLEU: 24.74 -> 27.12 | Helped |
| Use block proposals with annealed block size (ablation) | Raw Energy MH · Unconditional generation | GPT-2 sentence perplexity: 108.12 -> 43.21 | Helped |
| Use the raw rather than locally normalized energy parametrization (ablation) | MLM-based MH generation · Unconditional generation, BERT-base | GPT-2 sentence perplexity: 184.21 -> 125.42 | Helped |
| Use the raw rather than locally normalized energy parametrization (ablation) | MLM-based MH generation · WMT-14 De-En test | BLEU: 29.74 vs 30.12; 31.13 vs 30.86 | Mixed |
| Lower proposal temperature from 1.0 to 0.5 (ablation) | Locally Normalized Energy MH · WMT-14 De-En (20) | BLEU: 24.74 -> 27.23 | Helped |
| Lower proposal temperature from 1.0 to 0.5 (ablation) | Raw Energy MH · Unconditional generation | GPT-2 sentence perplexity: 108.12 -> 77.44 | Helped |
| Lower the nucleus boundary from 1.0 to 0.90 (ablation) | Locally Normalized Energy MH · WMT-14 De-En (20) | BLEU: 24.74 -> 27.35 | Helped |
| Anneal target energy temperature by 0.02 rather than 0.04 (ablation) | Raw Energy MH · WMT-14 De-En (20) | BLEU: 28.32 -> 29.91 | Helped |
| Increase target-temperature annealing strength from 0.02 to 0.06 (ablation) | Raw Energy MH · WMT-14 De-En (20) | BLEU: 29.91 -> 27.21 | Hurt |
| Use low-temperature masked-conditionals as a direct Gibbs sampler (ablation) | Degenerate Gibbs · WMT-14 De-En (20) | BLEU: 9.03 -> 26.12 | Mixed |
Closest papers by meaning
- Scaling up Masked Diffusion Models on Text (ICLR 2025 · similarity 0.90)
- Masked Diffusion Models are Secretly Time-Agnostic Masked Models and Exploit Inaccurate Categorical Sampling (ICLR 2025 · similarity 0.89)
- Theoretical Benefit and Limitation of Diffusion Language Model (NeurIPS 2025 · similarity 0.89)
- Accelerated Sampling from Masked Diffusion Models via Entropy Bounded Unmasking (NeurIPS 2025 · similarity 0.88)
- MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs (ICLR 2026 · similarity 0.88)
- A Probabilistic Perspective on Unlearning and Alignment for Large Language Models (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.