Research map / Architectures and efficiency / RNNs and normalization
Universal In-Context Approximation By Prompting Fully Recurrent Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that fully recurrent models (RNNs, LSTMs, GRUs, Linear RNNs, gated linear RNNs such as Mamba and Hawk/Griffin) can be universal in-context approximators, using a new compiler language LSRL that compiles programs into these architectures' weights.
Problem
Whether fully recurrent architectures (RNNs, LSTMs, GRUs, SSMs) can be prompted to approximate any function in-context with fixed weights, a property previously shown only for transformers.
Approach
Introduces LSRL, a programming language whose programs compile exactly to Linear RNN and Gated Linear RNN weights, and uses it to build prompt-based universal approximators for continuous functions and token-to-token maps; shows multiplicative gating yields more numerically stable conditional operators.
Key results
- Continuous-function approximation with Linear RNNs needs prompt length O(epsilon^-d_in), versus O(epsilon^(-10-14d-4d^2)) for the best known transformer construction.
- The ReLU-based f_ifelse conditional is numerically unstable after debranching; the token-to-token LSRL program got most tokens wrong under this implementation, while the symbolic backend produced correct outputs.
- Multiplicative-gate conditional operator is more robust to parameter noise than ReLU-based implementations.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace the original ReLU-based conditional operator with the optimized version (ablation) | Compiled LSRL token-function model · Parameter-noise robustness | numerical stability | Helped |
| Replace the original ReLU-based conditional operator with the step-based implementation (ablation) | Compiled LSRL model · Input noise | numerical stability | Helped |
| Replace the original ReLU-based conditional operator with the step-based implementation (ablation) | Compiled LSRL model · Parameter noise | numerical stability | Hurt |
| Use multiplicative gates for the conditional operator instead of ReLU-based implementations (ablation) | Compiled gated Linear RNN · Parameter-noise robustness | numerical stability | Helped |
Surprising findings
- The ReLU-based conditional operator, a standard construction, is numerically unstable once the program is compiled to a path-graph Linear RNN, and the token-level program gets most tokens wrong under it.
Limitations
- Constructive existence results only; cannot determine whether a given trained model has the property.
- No restrictions on the A matrix assumed, whereas practical SSMs use structured (e.g., diagonal) A.
- ReLU-based conditional operators are numerically unstable, so practical realizability is uncertain.
Experimental setup
Metrics: approximation error bound numerical robustness to parameter noise
Baselines: transformer universal in-context approximation (Petrov et al. 2023 style memoris Kajitsuka and Sato transformer construction
Builds on: RASP Tracr Linear RNN / SSM Hawk/Griffin GRU LSTM
Closest papers by meaning
- Transformers are Universal In-context Learners (ICLR 2025 · similarity 0.87)
- Prompting a Pretrained Transformer Can Be a Universal Approximator (ICML 2024 · similarity 0.87)
- Is attention required for ICL? Exploring the Relationship Between Model Architecture and In-Context Learning Ability (ICLR 2024 · similarity 0.87)
- Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding (NeurIPS 2025 · similarity 0.87)
- Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers (ICML 2024 · similarity 0.87)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.