Research map / Architectures and efficiency / RNNs and normalization

Expressivity of Neural Networks with Random Weights and Learned Biases

ICLR 2025 · Architectures and efficiency › RNNs and normalization · approach: Mechanistic circuit interpretability · 3 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Shows theoretically and numerically that feedforward and recurrent networks with fixed random weights and only learned hidden biases are universal approximators for static functions and finite-time dynamical system trajectories.

Problem

Whether universal function approximation and dynamical system approximation hold when only biases are learned while all weights remain random and frozen, relevant to neuroscience (tonic inputs) and LLM fine-tuning (bias/prefix methods).

Approach

Random frozen weights contain, with high probability, a sub-network that approximates the target; biases are set very negative to mask out non-participating units, so tuning biases alone realizes universal approximation (via a masking argument, extended to RNNs as an SLTH-over-units result).

Key results

• Bias-learned FNN with 3.2e4 hidden units achieves similar performance to fully-trained network on most of 7 image tasks (only significantly worse on KMNIST), using ~3.2e4 trained parameters versus ~2.5e7 for fully-trained.
• On MNIST, bias learning with 1e4 ReLU units reaches 0.934 test accuracy versus 0.919 for mask learning on the same random weights.
• Bias-learned RNNs need gain g>=1 on recurrent weights to learn a van der Pol oscillator; fully-trained networks are insensitive to gain.

Measured effects reported in the paper

Surprising findings

• Bias learning slightly outperforms fully-trained networks at very low parameter counts on Fashion MNIST.
• The theoretical width bound is extremely loose (e.g., ~8e12 units for a tiny MLP), far above what is needed in practice.

Limitations

• Dynamical system convergence only proven for finite-time trajectories.
• Theoretical width bounds are vastly overestimated and say little about practical scaling.
• Bias-learned networks need much larger hidden layers than fully trained networks to match performance; mask-learning comparison confounded by steepness schedule.

Experimental setup

Datasets: MNIST KMNIST Fashion MNIST Ethiopic-MNIST Vai-MNIST Osmanya-MNIST Kannada-MNIST Lorenz attractor van der Pol oscillator center-out reaching task

Benchmarks: Fashion MNIST MNIST KMNIST Ethiopic-MNIST Vai-MNIST Osmanya-MNIST Kannada-MNIST Lorenz attractor forecasting van der Pol oscillator center-out reaching

Metrics: validation accuracy validation error R^2 training loss task variance

Baselines: fully-trained networks mask learning parameter-matched fully-trained RNN

Builds on: Strong Lottery Ticket Hypothesis Malach et al. 2020 SLTH over units universal approximation theorems

Closest papers by meaning

• The impact of allocation strategies in subset learning on the expressive power of neural networks (ICLR 2025 · similarity 0.87)
• Improving equilibrium propagation without weight symmetry through Jacobian homeostasis (ICLR 2024 · similarity 0.86)
• Measuring and Controlling Solution Degeneracy across Task-Trained Recurrent Neural Networks (NeurIPS 2025 · similarity 0.85)
• How Uniform Random Weights Induce Non-uniform Bias: Typical Interpolating Neural Networks Generalize with Narrow Teachers (ICML 2024 · similarity 0.85)
• Learning dynamics in linear recurrent neural networks (ICML 2025 · similarity 0.85)
• From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.85)