Research map / Architectures and efficiency / Attention and transformers
L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a bipartite mutual information scaling law (sub-volume, power-law) for natural language and derives the L²M condition lower-bounding how fast a language model's history state must grow with context length, validated on Gaussian synthetic data and PG19 with Mamba and GPT2.
Problem
Lacking a theory of what model memory is needed to capture multi-token long-range dependencies in natural language, existing two-point mutual information analyses are argued to be incomplete for guiding long-context architecture design.
Approach
Uses bipartite mutual information between adjacent text segments, estimated with LLMs as density estimators (direct and vCLUB estimators), to show power-law growth; then bounds model capacity by history state dimension, giving the L²M condition that history state must scale at least as L^beta.
Key results
- Bipartite mutual information on PG19 measured with LLaMA 3.1 405B shows clear power-law (sub-volume) scaling over thousands of tokens, with estimators likely underestimating the exponent.
- Mamba KL divergence on synthetic Gaussian data collapses onto one curve when plotted against I^BP / dim(z) across sequence lengths 64 to 16,384 and model sizes 50M to 1.4B.
- Extrapolation suggests bipartite mutual information for a one-million-token sequence could exceed 60,000 nats, implying recurrent state dimensions approaching one million for low KL.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GPT2 instead of smaller Mamba or Mamba2 models | Synthetic Gaussian language modeling · Synthetic Gaussian, varying sequence lengths | average KL divergence per token | Helped |
| Increase sequence length for smaller Mamba and Mamba2 models | Mamba and Mamba2 · Synthetic Gaussian, varying sequence lengths | average KL divergence per token | Hurt |
| Increase Mamba or Mamba2 model size | Mamba and Mamba2 · Synthetic Gaussian, sequence length 4096 | average KL divergence per token | Helped |
| Scale recurrent state size relative to bipartite mutual information | Mamba · Synthetic Gaussian, sequence lengths 64 to 16,384 | average KL divergence per token | Helped |
| Use Mamba instead of GPT2 at early token positions | Mamba and GPT2 · PG19, 8192-token training sequences | position-wise conditional NLL | Helped |
| Use Mamba instead of GPT2 at later token positions | Mamba and GPT2 · PG19, 8192-token training sequences | position-wise conditional NLL | Hurt |
| Increase Mamba model size | Mamba · PG19, 8192-token training sequences | position-wise conditional NLL | Helped |
| Use bias-corrected direct estimation instead of uncorrected direct estimation (ablation) | Bipartite mutual information estimator · Natural language, varying sequence lengths | bipartite mutual information estimate | Mixed |
Surprising findings
- Two-point mutual information can mislead: a Markov-like identical-token distribution shows constant long-range two-point MI, and Gaussian families with identical two-point decay differ drastically in bipartite MI scaling (power-law vs logarithmic).
Limitations
- L2M condition is necessary but not sufficient and does not address reasoning or downstream task performance.
- Measured exponents from LLM-based estimators likely underestimate the true exponent; evaluation relies mainly on open-source models.
- Synthetic Gaussian validation may not capture all natural-language complexity.
Experimental setup
Datasets: PG19 Wikipedia synthetic multivariate Gaussian distributions
Benchmarks: PG19 Wikipedia synthetic Gaussian KL divergence test
Metrics: bipartite mutual information two-point mutual information per-token KL divergence position-wise negative log likelihood power-law exponent
Baselines: MINE InfoNCE K-NN estimators GPT2 Mamba Mamba2
Builds on: vCLUB Hilberg conjecture MINE InfoNCE Mamba
Closest papers by meaning
- Data Engineering for Scaling Language Models to 128K Context (ICML 2024 · similarity 0.89)
- Understanding LLM Behaviors via Compression: Data Generation, Knowledge Acquisition and Scaling Laws (NeurIPS 2025 · similarity 0.88)
- What is Wrong with Perplexity for Long-context Language Modeling? (ICLR 2025 · similarity 0.88)
- Short-Context Dominance: How Much Local Context Natural Language Actually Needs? (NeurIPS 2026 · similarity 0.88)
- M+: Extending MemoryLLM with Scalable Long-Term Memory (ICML 2025 · similarity 0.88)
- Beyond Attention: Breaking the Limits of Transformer Context Length with Recurrent Memory (AAAI 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.