Research map / Architectures and efficiency / Attention and transformers
How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a limited-domain pre-training setup with out-of-distribution testing to expose base-capability degradation in stateful sequence models, identifies full-sequence arbitrary selection as the design principle that avoids it, and validates it with Top-1 Element and Top-1 Chunk Selection architectures.
Problem
Existing sequence-modeling architecture studies evaluate under mixed-domain pre-training, which turns base-capability tests into in-distribution evaluations and hides architecture differences; it is unclear how architecture choices (e.g., stateful vs. attention) affect base capabilities of pre-train
Approach
Pre-train on a restricted set of domains (cc, c4) and test OOD on unseen domains (arxiv, github, stack) to reveal base-capability gaps; ablate architecture components to find that full-sequence visibility, real query-key relation computation and non-uniform attention distribution matter, then build a Top-1 element/chunk selection architecture satisfying this principle.
Key results
- Top-1 Chunk (Exact) reached 50.72 average few-shot score versus 50.43 for Transformer++ and 49.84 for Mamba-1 at 1.3B, 2k context.
- On retrieval tasks at 1.3B, Top-1 Element scored 68.23 on SWDE versus 36.09 for Mamba-1 and 37.98 for Mamba-2.
- At 100k sequence length, Top-1 Chunk (Approx.) reached 4.50x pre-training speed versus Transformer++ with 15.00 on S-NIAH 100k vs 15.80 for Transformer++.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use limited-domain pre-training with out-of-distribution evaluation instead of mixed-domain evaluation | sequence modeling architecture comparison · OOD language modeling | test loss | Mixed |
| Replace full data-dependent decay with shared decay or no explicit decay (ablation) | Mamba-1 and Mamba-2 · OOD language modeling | test performance | No clear effect |
| Remove convolution from the Decay None architecture (ablation) | Mamba · OOD language modeling | base capability | No clear effect |
| Remove GroupNorm (ablation) | Mamba-2 · OOD language modeling | base capability | No clear effect |
| Change position encoding among none, absolute, and rotary (ablation) | Transformer · OOD language modeling | test performance | No clear effect |
| Use AliBi position encoding (ablation) | Transformer · OOD language modeling | test performance | Hurt |
| Increase sliding-window size toward full-sequence visibility (ablation) | Transformer++ · OOD language modeling | test performance | Helped |
| Replace query-key relation scores with trainable random constants (ablation) | attention architecture · OOD language modeling | test performance | Hurt |
| Make attention distribution more non-uniform by decreasing softmax temperature (ablation) | self-attention · OOD language modeling | test performance | Helped |
| Select one element using full-sequence attention | Top-1 Element · OOD language modeling | test performance | Helped |
| Approximate chunk selection probabilities with a first-order linear function (ablation) | Top-1 Chunk · OOD language modeling | test performance | Hurt |
Surprising findings
- Under mixed-domain pre-training, different architectures (including stateful models) reach similar test loss at comparable pre-training loss, so the standard setting hides base-capability differences.
- Data-dependent decay, convolution, and position encoding (except ALiBi) affect only convergence speed rather than base capabilities.
Limitations
- Only language-modeling pre-training objective studied; conclusions limited to language models.
- Top-1 Chunk (Approx.) shows pre-training convergence and base capability degradation versus Transformer++.
- Experiments are at small scale (110M to 1.3B parameters).
Experimental setup
Datasets: SlimPajama AutoMathText RULER lm-evaluation-harness tasks (MMLU, ARC, BoolQ, PIQA, etc.) SWDE SQuAD FDA
Benchmarks: SlimPajama OOD test loss (arxiv, github, stack) MMLU OpenBookQA ARC-e ARC-c BoolQ RACE SIQA SCIQ HellaSwag COPA PIQA
Metrics: pre-training loss vs OOD test loss accuracy pre-train speed inference speed passkey retrieval accuracy
Baselines: Transformer++ Transformer++ (W=256) Mamba-1 Mamba-2 RWKV-5 RWKV-6 RWKV-7 DeltaNet Gated DeltaNet Based
Builds on: Transformer Mamba Gated DeltaNet RWKV-7 FlashAttention-2 Synthesizer
Closest papers by meaning
- How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers (NeurIPS 2024 · similarity 0.92)
- Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling (NeurIPS 2024 · similarity 0.87)
- Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (NeurIPS 2025 · similarity 0.86)
- Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities (NeurIPS 2025 · similarity 0.85)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.85)
- Scaling Laws Meet Model Architecture: Toward Inference-Efficient LLMs (ICLR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.