Research map / Architectures and efficiency / RNNs and normalization
Learning Useful Representations of Recurrent Neural Network Weight Matrices
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Learns self-supervised representations of LSTM weights by training encoders (mechanistic and functionalist probing) to emulate RNN behavior, and releases two RNN model-zoo datasets (formal languages and tiled sequential MNIST).
Problem
Learning useful representations of RNN weight matrices that support RNN analysis and downstream property prediction, which had no public datasets or methods before this work.
Approach
Compare mechanistic encoders (layer-wise statistics, flattened weights, parameter transformer, adapted DWSNet) with functionalist probing encoders, including a new interactive probing encoder whose probing inputs adapt to the RNN's previous outputs, trained via emulation of the RNN's rollouts; theoretically, interactive probing can be exponentially more efficient than non-interactive probing.
Key results
- On Formal Languages, interactive probing achieves the lowest emulation validation loss (0.015) versus 0.046 for DWSNet and 0.051 for Layer-Wise Statistics.
- For OOD task prediction on Formal Languages, pre-trained interactive probing representations significantly outperform other encoders and purely supervised models.
- On Sequential MNIST, both probing encoders reach emulation loss 0.017 versus 0.024 for DWSNet and Flattened Weights, while Parameter Transformer reaches 0.067.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use interactive instead of non-interactive probing | RNN weight encoder · Formal Languages validation | emulation loss: 0.023 -> 0.015 | Helped |
| Use probing encoders instead of mechanistic encoders | RNN weight representation learning · Sequential MNIST validation | emulation loss: 0.024 -> 0.017 | Helped |
| Use interactive instead of non-interactive probing | RNN weight encoder · Sequential MNIST validation | emulation loss: 0.017 -> 0.017 | No clear effect |
| Increase the number of parallel probing sequences (ablation) | Probing encoders · Formal Languages and Sequential MNIST | validation loss | Helped |
| Shorten the probing sequence below the task-sufficient length (ablation) | Interactive Probing · Formal Languages | validation loss | Hurt |
| Extend probing sequences beyond the task-sufficient length (ablation) | Interactive Probing · Formal Languages | training stability | Hurt |
| Use interactive probing rather than other encoders | Pre-trained RNN weight representations · Formal Languages OOD task prediction | task prediction performance | Helped |
| Use probing encoders rather than other architectures | RNN weight representations · Sequential MNIST OOD task prediction | task prediction performance | Helped |
| Use interactive rather than non-interactive probing in supervised training | RNN weight encoder · Sequential MNIST task prediction | task prediction performance | Hurt |
Surprising findings
- Parameter Transformer emulator classifies digits with high accuracy even when conditioned on low-performing RNNs, because it learns a rotation-invariant classifier instead of using the representation.
- In the supervised setting on Sequential MNIST, non-interactive probing performs best, while the interactive version does not, possibly due to limited supervised feedback.
Limitations
- Interactive probing suffers from training stability issues on some tasks.
- Experiments limited to relatively small LSTMs (hidden size 32).
- Functionalist encoders cannot distinguish functionally equivalent RNNs with different mechanisms.
Experimental setup
Datasets: Formal Languages model zoo (1000 LSTMs, 216 languages) Tiled Sequential MNIST model zoo (1000 LSTMs, rotated digits)
Benchmarks: Formal Languages model zoo Sequential MNIST model zoo
Metrics: self-supervised emulation validation loss downstream prediction loss (task, performance, generalizatio
Baselines: Layer-Wise Statistics Flattened Weights Parameter Transformer DWSNet Non-Interactive Probing supervised encoder trained from scratch
Builds on: DWSNet policy fingerprinting AdamW
Closest papers by meaning
- A unified theory of feature learning in RNNs and DNNs (ICML 2026 · similarity 0.90)
- Learning Recurrent Binary/Ternary Weights (ICLR 2019 · similarity 0.87)
- Eigenvalue Normalized Recurrent Neural Networks for Short Term Memory (AAAI 2020 · similarity 0.87)
- Decision-Guided Weighted Automata Extraction from Recurrent Neural Networks (AAAI 2021 · similarity 0.87)
- Learning Dynamics of RNNs in Closed-Loop Environments (NeurIPS 2025 · similarity 0.86)
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.