Research map / Architectures and efficiency / Attention and transformers
Language Models Need Inductive Biases to Count Inductively
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Trains RNNs, LSTMs, Transformers with five positional embeddings, SSMs and RWKV from scratch on synthetic counting tasks to test whether they can count inductively to OOD lengths.
Problem
Whether language models can learn to count inductively, i.e., generalize counting to cardinalities and sequence lengths beyond those seen in training, a capability underlying many claimed Transformer primitives.
Approach
Designs counting task formats (helper token, shifted start, modular, selective) that expose the number vocabulary without exposing OOD cardinalities, then compares architectures and positional embeddings, finding RNN/LSTM generalize while Transformers need specific PEs and modern RNNs underperform.
Key results
- Transformers with 1L or 2L fail to generalize inductively on shifted-start counting; only 4L RoPE generalizes (98.86% OOD at 4L) but fragile under larger length ratios.
- Traditional RNN and LSTM achieve 100% IND and OOD accuracy on helper token, shifted start and modular counting across layers.
- Modern RNNs (S4, Mamba, RWKV-v6) degrade on OOD; e.g., Mamba drops to 15.5% OOD on helper token 1L and RWKV-v6 to 0.0% on helper token + shifted start 1L.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use RoPE with 4 layers for shifted-start counting rather than NoPE with 4 layers | Transformer · Shifted Start OOD | accuracy: 46.1 -> 98.86 | Helped |
| Use SinePE or APE rather than NoPE for modular counting | Transformer · Modular (mod10) OOD | accuracy: 11.8 -> 100 | Helped |
| Use NoPE rather than RoPE for selective counting | Transformer · Selective OOD | accuracy: 52.5 -> 100 | Helped |
| Use LSTM rather than Mamba for modular counting | Recurrent models · Modular (mod10) OOD | accuracy: 11.5 -> 100 | Helped |
| Add shifted start to the helper-token task (ablation) | Transformer 2L RoPE · Helper Token + Shifted Start OOD | accuracy: 100 -> 78.7 | Hurt |
| Add BOS to NoPE (ablation) | Transformer NoPE · Modular (mod10) OOD | accuracy: 11.8 -> 10.1 | No clear effect |
| Add BOS to RoPE (ablation) | Transformer RoPE · Modular (mod10) OOD | accuracy: 11.8 -> 27.8 | Helped |
| Remove causal masking (ablation) | NoPE 1L Transformer on first-token recognition · first_tok_heterogeneous OOD | accuracy: 100 -> 0.0 | Hurt |
| Add a helper token to expose larger counts using another token (ablation) | Transformer shifted-start counting · Shifted Start OOD | accuracy | Mixed |
| Replace shifted positional embeddings with randomized positional embeddings (ablation) | Transformer · Counting OOD | accuracy | Hurt |
| Increase the OOD sequence length to four times the training length (ablation) | Transformer 4L RoPE shifted-start counting · Shifted Start OOD | accuracy | Hurt |
Surprising findings
- Transformer counting in-domain is not trivially achieved; 1L and 2L models often fail, contradicting treating counting as a primitive.
- Median performance across seeds is much lower than best for several Transformer PEs, indicating learnability depends on chance.
Limitations
- Results rely on best-of-five-seed and best-checkpoint reporting; medians are much lower for several configurations.
- Experiments limited to synthetic tasks with shallow models (1-4 layers) and fixed hyperparameter search.
- Mechanistic explanations for RoPE failures and BOS effects are partly left for future work.
Experimental setup
Datasets: synthetic counting sequences (vanilla+succession, helper token, shifted start, modular, selective, first_tok_homogeneous
Benchmarks: synthetic counting IND/OOD test sets
Metrics: accuracy (IND and OOD, best checkpoint over five seeds) median accuracy over five seeds
Baselines: Transformer NoPE Transformer SinePE Transformer APE Transformer RoPE Transformer SPE RNN LSTM S4 Mamba RWKV-v6
Builds on: RoPE Mamba RWKV shifted position IDs
Closest papers by meaning
- Understanding Counting Mechanisms in Large Language and Vision-Language Models (CVPR 2026 · similarity 0.89)
- Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers (ICML 2025 · similarity 0.88)
- On the Inductive Bias of Stacking Towards Improving Reasoning (NeurIPS 2024 · similarity 0.87)
- Phenomenal Yet Puzzling: Testing Inductive Reasoning Capabilities of Language Models with Hypothesis Refinement (ICLR 2024 · similarity 0.87)
- An Analysis for Reasoning Bias of Language Models with Small Initialization (ICML 2025 · similarity 0.87)
- Reasoning with Latent Thoughts: On the Power of Looped Transformers (ICLR 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.