Research map / Architectures and efficiency / Attention and transformers
TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
Yixing Li, Ruobing Xie, Zhen Yang, Xingwu Sun, Shuaipeng Li, Weidong Han, Zhanhui Kang, Di Wang, Yu Cheng
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use fine-grained 8-layer TransPoint cycle (V9) instead of layer-shared TransPoint [2048] (V1) (ablation) | TransMamba · validation loss at 21B tokens | validation loss: 3.356 -> 2.898 | Improved |
| make TransPoints layer-specific (V5) instead of layer-shared (V2) (ablation) | TransMamba · validation loss | validation loss: 3.297 -> 3.100 | Improved |
| cover a broad range of TransPoints (V8) instead of concentrated (V6) (ablation) | TransMamba · validation loss | validation loss: 3.135 -> 3.022 | Improved |
| refine to fine-grained schedule V9 instead of broad-range V8 (ablation) | TransMamba · validation PPL | PPL: 2.053 -> 1.813 | Improved |
| replace MLP converter with theoretical SSM Memory Converter (ablation) | TransMamba · training loss | training loss: 3.209 -> 3.173 | Improved |
| map attention with z of SSM (global z residual) instead of attention w/o z (ablation) | TransMamba · training loss | training loss: 3.39 -> 3.447 | Hurt |
| use TransMamba instead of Hybrid | 1.5B models · LongBench-v2 overall | accuracy: 35.79 -> 38.76 | Improved |
| use TransMamba instead of Hybrid | 1.5B models · PhoneBook | similarity: 26.63 -> 40.92 | Improved |
| use TransMamba instead of Transformer | 1.5B models · relative training time | time ratio: 1.00 -> 0.75 | Improved |
| infer with Transformer structure instead of the trained schedule (ablation) | TransMamba-400M · ARC-E | accuracy: 62.50 -> 27.27 | Hurt |
| infer with Hybrid structure instead of the trained schedule (ablation) | TransMamba-400M · OBQA | accuracy: 44.8 -> 45.0 | Improved |
| infer with Hybrid structure instead of the trained schedule (ablation) | TransMamba-400M · ARC-E | accuracy: 62.50 -> 52.20 | Hurt |
Closest papers by meaning
- The Mamba in the Llama: Distilling and Accelerating Hybrid Models (NeurIPS 2024 · similarity 0.91)
- Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (ICML 2024 · similarity 0.91)
- Can Mamba Learn How To Learn? A Comparative Study on In-Context Learning Tasks (ICML 2024 · similarity 0.90)
- DeciMamba: Exploring the Length Extrapolation Potential of Mamba (ICLR 2025 · similarity 0.90)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.90)
- Mamba Modulation: On the Length Generalization of Mamba Models (NeurIPS 2025 · similarity 0.89)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
