Research map / Architectures and efficiency / RNNs and normalization
Using Fast Weights to Attend to the Recent Past
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add fast associative memory to a ReLU RNN | Multi-level visual attention model · MNIST, 50 features | classification error rate: 12.95% -> 7.21% | Helped |
| Add fast associative memory to a ReLU RNN | Multi-level visual attention model · MNIST, 100 features | classification error rate: 1.55% -> 1.30% | Helped |
| Add fast associative memory to a ReLU RNN | Multi-level visual attention model · MNIST, 200 features | classification error rate: 1.10% -> 0.85% | Helped |
| Use fast associative memory instead of a ConvNet | MNIST classifier · MNIST, 50 features | classification error rate: 1.81% -> 7.21% | Hurt |
| Use fast associative memory instead of a ConvNet | MNIST classifier · MNIST, 200 features | classification error rate: 0.9% -> 0.85% | Helped |
| Add fast associative memory | Multi-level visual attention model · Facial expression recognition | test accuracy: 81.32 -> 86.34 | Helped |
| Replace the ConvNet with the fast-weights attention model | Facial expression recognition · Facial expression recognition | test accuracy: 88.23 -> 86.34 | Hurt |
| Apply layer normalization at every inner-loop iteration (ablation) | Fast associative memory · Fast-weight model training | robustness to learning-rate and decay hyperparameters | Helped |
| Use more inner-loop iterations (ablation) | Fast weights on associative retrieval · Associative retrieval | performance | No clear effect |
| Use inner-loop method 1 with sustained boundary conditions instead of method 2 with identity added to fast weights (ablation) | Fast associative memory · Inner-loop settling | performance | Helped |
Closest papers by meaning
- A Theory of Initialisation's Impact on Specialisation (ICLR 2025 · similarity 0.83)
- Sleep-Like Unsupervised Replay Improves Performance When Data Are Limited or Unbalanced (Student Abstract) (AAAI 2024 · similarity 0.83)
- Upweighting Easy Samples in Fine-Tuning Mitigates Forgetting (ICML 2025 · similarity 0.82)
- Improving Deep Learning Speed and Performance Through Synaptic Neural Balance (AAAI 2025 · similarity 0.82)
- Expressivity of Neural Networks with Random Weights and Learned Biases (ICLR 2025 · similarity 0.82)
- Learning Useful Representations of Recurrent Neural Network Weight Matrices (ICML 2024 · similarity 0.81)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.