Research map / Architectures and efficiency / RNNs and normalization
Inverse Approximation Theory for Nonlinear Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proves a Bernstein-type (inverse approximation) theorem showing that functionals stably approximable by nonlinear RNNs (tanh/hardtanh-like activations) must have exponentially decaying memory, and proposes stable reparameterizations (exp, softplus, LRU-style) to learn polynomial-memory targets.
Problem
Whether nonlinear recurrent activations lift the 'curse of memory' known for linear RNNs, i.e., which sequence-to-sequence functionals can be stably approximated by RNNs with nonlinear activations, and what memory structure they must have.
Approach
Defines a task-independent memory function for nonlinear functionals via the derivative response to Heaviside inputs, introduces a perturbation-based 'stable approximation' notion, and proves via Lyapunov/linearization arguments that stably approximable targets must have exponentially decaying memory; then reparameterizes recurrent weights (e.g., -exp(M), -softplus(M)) to keep eigenvalues negative
Key results
- Theorem: functionals stably approximated by tanh/hardtanh-like RNNs with uniformly bounded weights must have exponentially decaying memory, i.e., e^{beta t} M(H)(t) -> 0 for beta < beta_0.
- Linear functionals with exponential decaying memory admit stable approximation, while polynomial decaying memory targets show discontinuous perturbation error E(beta) at beta=0 for standard RNNs.
- Exp and softplus reparameterizations allow linear RNNs to stably approximate linear functionals with polynomial decaying memory.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use tanh RNNs to approximate a target with polynomial-decaying memory | tanh RNN · Target with polynomial-decaying memory | perturbation stability | Hurt |
| Use an RNN to approximate a target with exponential-decaying memory | RNN · Target with exponential-decaying memory | epochs to approximation: typically exponential decaying memory achieves the approximation at 10 epochs | Helped |
| Reparameterize recurrent weights with the exponential function | Linear RNN · Linear functional with polynomial-decay memory | stable approximation | Helped |
| Reparameterize recurrent weights with the softplus function | Linear RNN · Linear functional with polynomial-decay memory | stable approximation | Helped |
Surprising findings
- Adding nonlinear activations does not remove the curse of memory: nonlinear RNNs still require exponentially decaying memory for stable approximation.
- Approximation at polynomial-memory targets takes around 1000 epochs versus about 10 epochs for exponential-memory targets.
Limitations
- Theorem restricted to activations in a specific class (locally linear or locally tanh near zero, bounded monotone Lipschitz) and to continuous, causal, regular, time-homogeneous functionals.
- Stated for uniformly bounded weights and assumes limit (not limsup) in m for simplicity; GRU/LSTM analysis deferred.
- Numerical verification is limited to small synthetic settings and filtering; the reparameterization is shown mainly for linear functionals.
Experimental setup
Datasets: IMDB movie reviews
Benchmarks: IMDB sentiment classification (memory querying)
Metrics: perturbation error E(beta) queried memory function approximation error
Baselines: linear RNN with direct parameterization tanh RNN LSTM
Builds on: Bernstein-type linear RNN result (Li et al.) Riesz representation theorem LRU
Closest papers by meaning
- On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis (ICLR 2021 · similarity 0.90)
- On the approximation properties of recurrent encoder-decoder architectures (ICLR 2022 · similarity 0.89)
- State-space models with layer-wise nonlinearity are universal approximators with exponential decaying memory (NeurIPS 2023 · similarity 0.89)
- Learning and Generalization in RNNs (NeurIPS 2021 · similarity 0.88)
- Recurrent neural networks: vanishing and exploding gradients are not the end of the story (NeurIPS 2024 · similarity 0.87)
- StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.