Research map / Architectures and efficiency / Attention and transformers

InfoFlow: A Framework for Multi-Layer Transformer Analysis

NeurIPS 2026 · Architectures and efficiency › Attention and transformers · approach: Transformer expressivity and learnability · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes InfoFlow, a theoretical framework tracking information sets per token and layer with parameter-cost laws, motivated by a depth separation (single-layer needs exponential parameters vs. two-layer O(ε^-1)) and softmax max-position retrieval limits.

Problem

Approximation theory for multi-layer Transformers is largely open; single-layer results do not extend directly, and there is no tractable way to predict approximation rates of multi-layer models before training.

Approach

Replace hidden representations with discrete information sets I(t,l) of accessible input positions, with three propagation modes (max-position retrieval, global aggregation, specific-position aggregation) each assigned a parameter-cost exponent, and estimate approximation rates via a Number of Comparison of the target.

Key results

• Two-layer one-head Transformers approximate the target with O(ε^-1) parameters, while single-layer models require Ω(ε^-k) with k growing linearly in T (theorem statement).
• On the intrinsic dimension target, (D,D) head configurations reach validation NMSE around 1e-5 while (D-1,D) or (D,D-1) reach 1e-3 to 1e-2.
• For the triangle-center target, NMSE exceeds 0.85 at T=16 and approaches 1.0 for T≥32 across all three tested configurations.

Surprising findings

• Increasing the number of heads or embedding dimension does not improve approximation of the triangle-center target within the tested range.

Limitations

• Parameter cost law ignores the role of attention weight matrix rank.
• Propagation modes are motivated by synthetic tasks; real-world relevance needs validation.
• Framework applies only to retrieval-type targets whose active index set does not grow with T.

Experimental setup

Datasets: synthetic retrieval targets (intrinsic dimension target, triangle-center target)

Metrics: normalized mean squared error (NMSE)

Builds on: Transformer universal approximation single-layer Transformer approximation rates induction head analysis Transformer Circuits

Closest papers by meaning

• Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators? (ICLR 2024 · similarity 0.88)
• A unified framework for establishing the universal approximation of transformer-type architectures (NeurIPS 2025 · similarity 0.88)
• Approximation Rate of the Transformer Architecture for Sequence Modeling (NeurIPS 2024 · similarity 0.87)
• kNN Attention Demystified: A Theoretical Exploration for Scalable Transformers (ICLR 2025 · similarity 0.87)
• Token Statistics Transformer: Linear-Time Attention via Variational Rate Reduction (ICLR 2025 · similarity 0.87)
• DenseFormer: Enhancing Information Flow in Transformers via Depth Weighted Averaging (NeurIPS 2024 · similarity 0.86)