Research map / Architectures and efficiency / RNNs and normalization
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention
Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Train on MNIST then Fashion-MNIST sequentially instead of evaluating after MNIST-only training | Two-hidden-layer feedforward NN · MNIST test set | accuracy: 97% -> 45% | Hurt |
| Train on MNIST then Fashion-MNIST sequentially rather than jointly | Two-hidden-layer feedforward NN · Fashion-MNIST test set | accuracy: 87% -> 85% | Hurt |
| Add the norm of the error vector to attention-weight analysis | Dual-form attention analysis · Image classification | consistency of attention patterns | No clear effect |
| Use total per-class attention scores from later layers rather than raw input-layer scores | Dual-form attention analysis · MNIST test set | agreement of attention argmax with model output: 49.5 $\pm$ 1.5 -> 60.1 $\pm$ 4.5 | Improved |
| Train jointly on MNIST and Fashion-MNIST, allowing cross-task attention | Two-hidden-layer feedforward NN · MNIST and Fashion-MNIST | cross-task attention | Improved |
| Train successively on MNIST then Fashion-MNIST without masking later-task training patterns | Two-hidden-layer feedforward NN · Continual learning on MNIST and Fashion-MNIST | interference from later-task patterns | Hurt |
Closest papers by meaning
- What can a Single Attention Layer Learn? A Study Through the Random Features Lens (NeurIPS 2023 · similarity 0.87)
- In-Context Linear Regression Demystified: Training Dynamics and Mechanistic Interpretability of Multi-Head Softmax Attention (ICML 2025 · similarity 0.86)
- Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networks (NeurIPS 2025 · similarity 0.86)
- Training Dynamics of In-Context Learning in Linear Attention (ICML 2025 · similarity 0.86)
- Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks (ICML 2024 · similarity 0.86)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
