Research map / Large language models / Language, safety and interpretability
Residual Context Diffusion Language Models
Yuezhou Hu, Harman Singh, Monishwaran Maheswaran, Haocheng Xi, Coleman Richard Charles Hooper, Jintao Zhang, Aditya Tomar, Michael W. Mahoney, Sewon Min, Mehrdad Farajtabar, Kurt Keutzer, Amir Gholami and 1 more
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add residual context diffusion | SDAR-8B-b64 SeqD · AIME24 | accuracy (%): 7.08 -> 18.75 | Improved |
| add residual context diffusion | SDAR-8B-b64 SeqD · AIME25 | accuracy (%): 9.79 -> 16.04 | Improved |
| add residual context diffusion | SDAR-8B-b32 SeqD · MATH500 | accuracy (%): 65.80 -> 76.20 | Improved |
| add residual context diffusion | LLaDA SeqD · MinervaMath | accuracy (%): 31.10 -> 37.00 | Improved |
| add residual context diffusion | LLaDA SeqD · GSM8K | accuracy (%): 75.74 -> 78.09 | Improved |
| add residual context diffusion | SDAR-4B-b32 SeqD · GSM8K | accuracy (%): 81.73 -> 84.91 | Improved |
| train baseline longer (8 epochs) instead of adding RCD (ablation) | SDAR-8B-b64 SeqD · MATH500 | accuracy (%): 68.00 -> 74.40 | Mixed |
| Dirichlet init instead of 1.7B reference-model init (ablation) | RCD SDAR-8B-b64 · MATH500 | accuracy (%): 74.2 -> 77.4 | Improved |
| zero init instead of 4B reference-model init (ablation) | RCD SDAR-8B-b64 · MATH500 | accuracy (%): 71.8 -> 75.6 | Improved |
| add RCD residual vs Loopholing hidden-state injection (ablation) | SDAR-4B-b64 · GSM8K | accuracy (%): NA -> 86.35 | Improved |
| add normalized-entropy weighting of residual (ablation) | RCD interpolation · SDAR-8B-b64 untrained draft | Pareto frontier accuracy vs tokens per step | Improved |
| add RCD under matched throughput (Fastdllm) | LLaDA SeqD · GSM8K | accuracy (%): 76.12 -> 78.70 | Improved |
Closest papers by meaning
- Fast and Fluent Diffusion Language Models via Convolutional Decoding and Rejective Fine-tuning (NeurIPS 2025 · similarity 0.91)
- CORE: Context-Robust Remasking for Diffusion Language Models (ICML 2026 · similarity 0.91)
- dKV-Cache: The Cache for Diffusion Language Models (NeurIPS 2025 · similarity 0.90)
- UltraLLaDA: Scaling the Context Length to 128K for Diffusion Large Language Models (ICLR 2026 · similarity 0.90)
- SimSD: Simple Speculative Decoding in Diffusion Language Models (NeurIPS 2026 · similarity 0.90)
- Multi-Token Residual Prediction (NeurIPS 2026 · similarity 0.90)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
