Research map / Architectures and efficiency / RNNs and normalization
Implicit Language Models are RNNs: Balancing Parallelization and Expressivity
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes implicit SSMs and implicit transformers defined by fixed points of self-iterated state-space/attention blocks, trained with phantom gradients and a curriculum, showing improved state tracking and language modeling up to 1.3B parameters on 207B tokens.
Problem
Transformers and linear SSMs cannot track state or recognize regular languages such as the S5 word problem, while RNNs can but lack parallelization; the trade-off between parallelism and expressivity needs resolving for language models.
Approach
Define the sequence model implicitly via fixed points of a Mamba2 or Llama block that self-iterates along depth with input injection, enabling non-linear, non-diagonal state transitions; train with phantom gradients and a bounded-then-free self-iteration curriculum so most computation remains parallel.
Key results
- Implicit Llama (32+4) 1.3B reaches 7.24 D-Pile perplexity vs 7.99 for explicit Llama† trained on the same data, with average downstream accuracy 0.5892 vs 0.5589.
- Implicit Mamba2 (24+4) 1.3B reaches 7.70 D-Pile perplexity vs 8.28 for Mamba2* on the same 207B tokens, with average accuracy 0.5620 vs 0.5531.
- Implicit Mamba2 learns the S5-mixture word problem with out-of-distribution generalization at p=0.5 using about 8 training-time self-iterations, versus 256 steps for an RNN, while explicit SSMs stay at chance OOD.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use implicit Mamba2 (24+4) instead of Mamba2* | Mamba2 at 1.3B · D-Pile test | perplexity: 8.28 -> 7.70 | Helped |
| Use implicit Mamba2 (24+4) instead of Mamba2* | Mamba2 at 1.3B · HellaSwag | accuracy: 0.5939 -> 0.6267 | Helped |
| Use implicit Llama (32+4) instead of Llama† | Llama at 1.3B · D-Pile test | perplexity: 7.99 -> 7.24 | Helped |
| Use implicit Llama (32+4) instead of Llama† | Llama at 1.3B · HellaSwag | accuracy: 0.5936 -> 0.6583 | Helped |
| Increase implicit Mamba2 iterations from 4+1 to 24+4 (ablation) | Mamba2 at 1.3B · D-Pile test | perplexity: 7.97 -> 7.70 | Helped |
| Increase implicit Llama iterations from 4+1 to 32+4 (ablation) | Llama at 1.3B · D-Pile test | perplexity: 7.66 -> 7.24 | Helped |
| Increase implicit Mamba2 iterations from 4+1 to 24+4 (ablation) | Mamba2 at 1.3B · HellaSwag | accuracy: 0.6136 -> 0.6267 | Helped |
| Increase implicit Llama iterations from 4+1 to 32+4 (ablation) | Llama at 1.3B · HellaSwag | accuracy: 0.6397 -> 0.6583 | Helped |
| Use implicit Mamba2 rather than explicit Mamba2 | Mamba2 language models · Winogrande | accuracy | Hurt |
| Use implicit Mamba2 rather than explicit Mamba2 | Mamba2 language models · OpenbookQA | accuracy | Hurt |
| Train implicit Mamba2 with at least 8 self-iterations (ablation) | Implicit Mamba2 on mixed A5 word problems · OOD accuracy at p=0.5 | accuracy | Helped |
| Train on a 10% mixture of hard tokens (ablation) | Implicit Mamba2 on mixed A5 word problems · OOD evaluation at p=0.5 | accuracy | Helped |
Surprising findings
- Only approximate fixed-point convergence during training is needed, and a small number (about 8) of training-time self-iterations suffices for OOD state tracking.
- Simultaneous-mode training and sequential-mode generation agree on 97.6% of tokens for the 1.3B implicit Mamba2 at constant memory.
Limitations
- Self-iteration adds compute overhead (about 2-4x wall-clock time), which may only amortize on state-tracking-heavy tasks.
- Implicit models underperform on Winogrande and OpenbookQA at several scales.
- Models were trained on 207B tokens of D-Pile, not the 300B tokens of the original baselines, so comparisons with published baselines are not fully matched.
Experimental setup
Datasets: D-Pile CatbAbI BAbI synthetic word problems (S5, A5 mixtures)
Benchmarks: LAMBADA HellaSwag PIQA ARC-Easy ARC-Challenge WinoGrande OpenbookQA D-Pile test perplexity CatbAbI
Metrics: perplexity accuracy out-of-distribution accuracy relative wall-clock time
Baselines: Mamba2 Llama Mamba2* Llama† Universal Transformer
Builds on: Mamba2 Llama Deep equilibrium models Phantom gradient Illusion of state
Closest papers by meaning
- The Expressive Capacity of State Space Models: A Formal Language Perspective (NeurIPS 2024 · similarity 0.89)
- M$^2$RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling (NeurIPS 2026 · similarity 0.88)
- Language Models Are Implicitly Continuous (ICLR 2025 · similarity 0.88)
- Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models (ICML 2026 · similarity 0.87)
- Longhorn: State Space Models are Amortized Online Learners (ICLR 2025 · similarity 0.87)
- Skip a Layer or Loop It? Learning Program-of-Layers in LLMs (ICML 2026 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.