Research map / Architectures and efficiency / Attention and transformers
B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes B'MOJO, a stackable hybrid SSM-attention module combining fading (SSM state) and eidetic memory selected by an innovation test, evaluated on recall tasks, language modeling up to 1.4B, and length generalization.
Problem
Existing architectures represent past data either eidetically over a finite context (Transformers) or only as fading state (SSMs); hybrids do not seamlessly modulate the two or extend eidetic span, limiting transductive inference and recall over long sequences.
Approach
Combine an SSM fading-memory state with an eidetic memory populated by an "innovation selection" test that stores tokens whose prediction error from the fading state is high, and attend over the sliding window plus these selected tokens, so eidetic span is not limited by recency.
Key results
- At 1.4B, BMoJo (Fading + Eidetic) reaches 2.26 pre-training log-perplexity versus 2.37 for Mamba and 2.27 for Mistral, with zero-shot short-context average 49.1 vs 48.6 for Mamba and 51.9 for Mistral.
- On long-context SWDE at 370M, BMoJo (Fading + Eidetic) scores 17.91 vs 7.38 for Mamba and 47.16 for Mistral; at 790M BMoJo (Fading) reaches 22.59 vs Mamba's 17.37.
- The paper reports B'MOJO is up to 10% faster to train than similarly sized Transformers and SSMs (the introduction states 15%).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add fading memory to the vanilla hybrid architecture (ablation) | B'MOJO-F · Language modeling pre-training, all tested scales | perplexity | Helped |
| Add eidetic memory to B'MOJO-F (ablation) | B'MOJO · Language modeling pre-training, all tested scales | perplexity | Helped |
| Add eidetic memory to BMoJo (Fading) (ablation) | B'MOJO · SWDE (370M) | accuracy: 15.56 -> 17.91 | Helped |
| Add eidetic memory to BMoJo (Fading) (ablation) | B'MOJO · PG-19 (370M) | log-ppl: 3.05 -> 3.04 | Helped |
| Add eidetic memory to BMoJo (Fading) (ablation) | B'MOJO · Scrolls-QAsper (370M) | accuracy: 10.29 -> 9.02 | Hurt |
| Add eidetic memory to BMoJo (Fading) (ablation) | B'MOJO · Scrolls-NarraQA (370M) | accuracy: 7.48 -> 5.92 | Hurt |
| Increase the number of eidetic memory tokens (ablation) | B'MOJO · Multi-Query Associative Recall | accuracy | Helped |
Surprising findings
- On the 370M PG-19 long-context evaluation the full-attention Mistral baseline has the best SWDE score (47.16), yet on the 790M scale the eidetic variant underperforms the fading-only variant on Scrolls tasks.
- Models trained on longer contexts underperform models trained on shorter ones when evaluated on fewer tokens.
Limitations
- Experiments limited to scales up to 1.4B parameters, so scaling behavior beyond that is unverified.
- Mamba-like length generalization degrades with more compute, and B'MOJO may inherit this weakness.
- Short-context benchmark accuracy lags the full-attention Mistral baseline, with a larger gap at 1.4B.
Experimental setup
Datasets: MQAR synthetic Selective Copying Induction Heads Noisy MQAR Fuzzy MQAR PG-19 SWDE Scrolls-QAsper Scrolls-NarraQA LAMBADA HellaSwag PIQA
Benchmarks: MQAR Selective Copying Induction Heads Noisy MQAR Fuzzy MQAR LAMBADA HellaSwag PIQA ARC-E ARC-C WinoGrande PG-19
Metrics: accuracy log-perplexity wall-clock training time throughput
Baselines: Mistral (Full-Attention) Mamba Hybrid (Sliding Attention + SSM) GPT2 Pythia 160m
Builds on: Mamba Mistral Stochastic Realization Theory Gated Linear Attention
Closest papers by meaning
- xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference (ICML 2025 · similarity 0.87)
- Rethinking Expressivity and Efficiency in Test-Time Training (NeurIPS 2026 · similarity 0.87)
- Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation (NeurIPS 2025 · similarity 0.87)
- Repeat After Me: Transformers are Better than State Space Models at Copying (ICML 2024 · similarity 0.86)
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.86)
- Memory Layers at Scale (ICML 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.