Research map / Architectures and efficiency / RNNs and normalization

How Long Does Infinite Width Last? Signal Propagation in Long-Range Linear Recurrences

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Brain-inspired recurrent architectures · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Derives exact finite-width formulas for signal energies in linear recurrent units and RNNs with complex Gaussian weights, identifying subcritical, critical (t~sqrt n) and supercritical depth-width regimes where infinite-width theory breaks down.

Problem

When does the infinite-width approximation of signal propagation remain accurate in linear recurrent models as recurrent depth t grows jointly with width n? Existing theory relies on infinite-width limits and gives only crude bounds on finite-width signal growth.

Approach

Exact closed-form finite-width signal energies for linear RNNs and LRUs under complex Gaussian initialization, obtained by a Wick expansion reducing the trace moment to counting cycles of commutators of permutations (linked to Hultman numbers), revealing a sqrt(n) critical depth scale.

Key results

• Exact finite-width formula: Q_{n,k} = (F+_{k+2}(n) - F-_{k+2}(n)) / (n^{k+1}(k+1)(k+2)), and S_{n,t} = sum_k Q_{n,k}.
• In the critical regime k ~ c sqrt(n), Q_{n,k} converges to (2/c^2) sinh(c^2/2), exceeding the infinite-width value of 1.
• Supercritical regime sqrt(n) << t << n gives S_{n,t} ~ (n^2/t^3) exp(t^2/(2n)), so signal explodes once depth exceeds sqrt(n).

Surprising findings

• Recurrent models show finite-width signal explosion at depth sqrt(n), whereas feedforward networks with independent weights only deviate at depth of order n.
• ReLU and LeakyReLU recurrences show faster signal growth than linear recurrences in numerical experiments.

Limitations

• Exact results are only for complex Gaussian weights; real-weight case gives only a lower bound.
• Nonlinear recurrences are studied only numerically, without theory.
• Analysis is at random initialization and does not cover training dynamics.

Experimental setup

Metrics: signal energy Q_{n,k} signal energy S_{n,t}

Baselines: infinite-width signal propagation theory Glorot initialization feedforward networks with independent weights

Builds on: Glorot initialization linear recurrent unit Wick theorem Hultman numbers

Closest papers by meaning

• Revisiting Glorot Initialization for Long-Range Linear Recurrences (NeurIPS 2025 · similarity 0.87)
• On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling (NeurIPS 2025 · similarity 0.86)
• Understanding and Improving Length Generalization in Recurrent Models (ICML 2025 · similarity 0.86)
• How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models (NeurIPS 2026 · similarity 0.85)
• On the Diminishing Returns of Width for Continual Learning (ICML 2024 · similarity 0.84)
• Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues (ICML 2024 · similarity 0.84)