AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model

Yixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Di Wang, Yu Cheng

AAAI 2026 · Architectures and efficiency › Attention and transformers · approach: Mamba architectures · 20 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
use fine-grained 8-layer TransPoint cycle (V9) instead of layer-shared TransPoint [2048] (V1) (ablation)TransMamba · validation loss at 21B tokensvalidation loss: 3.356 -> 2.898Improved
make TransPoints layer-specific (V5) instead of layer-shared (V2) (ablation)TransMamba · validation lossvalidation loss: 3.297 -> 3.100Improved
cover a broad range of TransPoints (V8) instead of concentrated (V6) (ablation)TransMamba · validation lossvalidation loss: 3.135 -> 3.022Improved
refine to fine-grained schedule V9 instead of broad-range V8 (ablation)TransMamba · validation PPLPPL: 2.053 -> 1.813Improved
replace MLP converter with theoretical SSM Memory Converter (ablation)TransMamba · training losstraining loss: 3.209 -> 3.173Improved
map attention with z of SSM (global z residual) instead of attention w/o z (ablation)TransMamba · training losstraining loss: 3.39 -> 3.447Hurt
use TransMamba instead of Hybrid1.5B models · LongBench-v2 overallaccuracy: 35.79 -> 38.76Improved
use TransMamba instead of Hybrid1.5B models · PhoneBooksimilarity: 26.63 -> 40.92Improved
use TransMamba instead of Transformer1.5B models · relative training timetime ratio: 1.00 -> 0.75Improved
infer with Transformer structure instead of the trained schedule (ablation)TransMamba-400M · ARC-Eaccuracy: 62.50 -> 27.27Hurt
infer with Hybrid structure instead of the trained schedule (ablation)TransMamba-400M · OBQAaccuracy: 44.8 -> 45.0Improved
infer with Hybrid structure instead of the trained schedule (ablation)TransMamba-400M · ARC-Eaccuracy: 62.50 -> 52.20Hurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy