AI Paper Atlas

Research map / Architectures and efficiency / RNNs and normalization

The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention

Kazuki Irie, Róbert Csordás, Jürgen Schmidhuber

ICML 2022 · Architectures and efficiency › RNNs and normalization · approach: Representation geometry and collapse · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Train on MNIST then Fashion-MNIST sequentially instead of evaluating after MNIST-only trainingTwo-hidden-layer feedforward NN · MNIST test setaccuracy: 97% -> 45%Hurt
Train on MNIST then Fashion-MNIST sequentially rather than jointlyTwo-hidden-layer feedforward NN · Fashion-MNIST test setaccuracy: 87% -> 85%Hurt
Add the norm of the error vector to attention-weight analysisDual-form attention analysis · Image classificationconsistency of attention patternsNo clear effect
Use total per-class attention scores from later layers rather than raw input-layer scoresDual-form attention analysis · MNIST test setagreement of attention argmax with model output: 49.5 $\pm$ 1.5 -> 60.1 $\pm$ 4.5Improved
Train jointly on MNIST and Fashion-MNIST, allowing cross-task attentionTwo-hidden-layer feedforward NN · MNIST and Fashion-MNISTcross-task attentionImproved
Train successively on MNIST then Fashion-MNIST without masking later-task training patternsTwo-hidden-layer feedforward NN · Continual learning on MNIST and Fashion-MNISTinterference from later-task patternsHurt

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy