Research map / Architectures and efficiency / RNNs and normalization
The impact of allocation strategies in subset learning on the expressive power of neural networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Theoretical framework (match probability) showing that how a fixed number of learnable weights is allocated across linear RNN/FFN and shallow ReLU networks determines expressivity, with more distributed allocations generally better.
Problem
How should a limited number of learnable weights be allocated across a neural network so that the network can best express a target function, i.e., what allocation strategies maximize expressive power under subset learning?
Approach
Defines a student-teacher benchmark, the match probability of an allocation (probability that a student with random fixed weights and r learnable weights can exactly match teacher outputs on m samples), and derives conditions under which allocations are maximal (MP=1) or minimal (MP=0) for linear RNN encoders, decoders, recurrent weights and linear FFN layers; heuristics extend to shallow ReLU net
Key results
- For linear RNNs, allocations where every row has exactly Tb learnable weights (or 0) are proven maximal (MP=1), while rows with more than Tb learnable weights make the allocation minimal (MP=0).
- In a small LRNN example (n=2, T=2), match probability was 0.83 for off-diagonal and 0.74 for diagonal allocations.
- Numerical simulations show MP increases as allocations use more rows and approaches 1 for large networks; on MNIST with a shallow ReLU net (n=1000), MP was constant 1 when allocation used more than 40 rows.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Distribute learnable recurrent weights so each used row has Tb weights | Linear RNN recurrent weights · LRNN recurrent-weight allocations | match probability | Helped |
| Concentrate more than Tb learnable weights in a recurrent-weight row | Linear RNN recurrent weights · LRNN recurrent-weight allocations | match probability | Hurt |
| Distribute learnable recurrent weights so each used column has Td weights | Linear RNN recurrent weights · LRNN recurrent-weight allocations | match probability | Helped |
| Concentrate more than Td learnable weights in a recurrent-weight column | Linear RNN recurrent weights · LRNN recurrent-weight allocations | match probability | Hurt |
| Use the off-diagonal allocation instead of the diagonal allocation | Linear RNN with n=2, T=2, b=1, d=1, m=2 · LRNN recurrent weights | match probability: 0.74 -> 0.83 | Helped |
| Use more rows for the learnable recurrent-weight allocation | Linear RNN · LRNN recurrent weights | match probability | Helped |
| Increase hidden-state network size at the same percentage of allocated rows | Linear RNN · LRNN recurrent weights | match probability | Helped |
| Give every decoder row exactly m learnable weights | Linear RNN decoder · LRNN decoder allocations | match probability: 0 -> 1 | Helped |
| Distribute encoder weights without exceeding the row and column limits | Linear RNN encoder · LRNN encoder allocations | match probability: 0 -> 1 | Helped |
| Distribute learnable weights across more rows | Three-layer linear feed-forward network · Linear FFN allocation simulations | match probability | Helped |
| Distribute learnable input-layer weights across more rows | Shallow ReLU network · Shallow ReLU allocation simulations | match probability | Helped |
| Distribute learnable weights across more rows | Shallow ReLU network · MNIST | match probability | Helped |
Surprising findings
- Allocations of learnable weights that are concentrated in few rows can have zero expressive power even when the total number of learnable weights is fixed, with a sharp transition between minimal and maximal allocations.
- Even in the non-linear recurrent case, maximal allocations exist and for large networks MP for all allocations tends to approach 1.
Limitations
- Focuses on expressivity only, not generalization or optimization.
- Relies on simplifying assumptions such as i.i.d. Gaussian weights, linear independence of inputs, and linear architectures.
- Many results are for linear models; the ReLU extension is largely heuristic.
Experimental setup
Datasets: MNIST
Benchmarks: MNIST
Metrics: match probability
Builds on: lottery ticket hypothesis neural reprogramming
Closest papers by meaning
- Expressivity of Neural Networks with Random Weights and Learned Biases (ICLR 2025 · similarity 0.87)
- Scaling Laws and Tradeoffs in Recurrent Networks of Expressive Neurons (NeurIPS 2026 · similarity 0.86)
- From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks (ICLR 2025 · similarity 0.86)
- How connectivity structure shapes rich and lazy learning in neural circuits (ICLR 2024 · similarity 0.86)
- Improving Deep Learning Speed and Performance Through Synaptic Neural Balance (AAAI 2025 · similarity 0.86)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.