AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

DiffuMamba: High-Throughput Diffusion LMs with Mamba Backbone

Vaibhav Singh, Oleksiy Ostapenko, Pierre-Andre Noel, Eugene Belilovsky, Torsten Scholak

ICML 2026 · Architectures and efficiency › Attention and transformers · approach: State space and recurrent sequence models · 0 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
add one attention mixer every N=5 Mamba mixers (hybrid)DiffuMamba · DCLM validation (Quokka budget)validation perplexity: 21.41 -> 20.17Improved
replace attention mixers with bidirectional MambaDiffuTran · DCLM validation (Quokka budget)validation perplexity: 22.72 -> 21.41Improved
use hybrid Mamba-attention instead of pure attentionDiffuTran 1.3B · Zero-shot PPL avgperplexity: 55.43 -> 51.18Improved
use Mamba backbone instead of attention at small scaleDiffuTran 240M · Zero-shot PPL avgperplexity: 81.79 -> 78.11Mixed
use Mamba instead of attention at 0.5BDiffuTran 0.5B · Zero-shot PPL avgperplexity: 61.29 -> 66.01Hurt
Mamba-based backbone for downstream zero-shot tasksDiffuTran 1.3B · Downstream average accuracyaccuracy: 33.81 -> 38.12Improved
hybrid versus pure Mamba on downstream tasksDiffuMamba 1.3B · Downstream average accuracyaccuracy: 38.12 -> 38.24No clear effect
evaluate at double training length (L=2048 vs 1024)DiffuTran 1.3B · Length generalizationavg zero-shot PPL: 32.20 -> 56.57Hurt
evaluate at double training length (L=2048 vs 1024)DiffuMamba-H 1.3B · Length generalizationavg zero-shot PPL: 28.88 -> 28.53No clear effect
evaluate at double training length (L=2048 vs 1024)DiffuMamba-H 1.3B · Length generalizationMAUVE: 0.744 -> 0.796Improved
add hybrid interleaving (N=5) to pure attention (ablation)DiffuTran 0.5B · Hybrid interleaving ablationMAUVE: 0.672 -> 0.719Improved
add hybrid interleaving (N=5) to pure Mamba (ablation)DiffuMamba 0.5B · Hybrid interleaving ablationMAUVE: 0.674 -> 0.719Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy