Research map / Architectures and efficiency / RNNs and normalization
Input-Cell Attention Reduces Vanishing Saliency of Recurrent Neural Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add input-cell attention over current and previous inputs | LSTM · Earlier Box (synthetic) | weighted Jaccard: 0.048 -> 0.103 | Helped |
| Add input-cell attention over current and previous inputs | LSTM · Middle Box (synthetic) | weighted Jaccard: 0.048 -> 0.124 | Helped |
| Add input-cell attention over current and previous inputs | LSTM · Latter Box (synthetic) | weighted Jaccard: 0.045 -> 0.110 | Helped |
| Add input-cell attention over current and previous inputs | LSTM · Mixed Boxes (synthetic) | weighted Jaccard: 0.060 -> 0.104 | Helped |
| Add input-cell attention over current and previous inputs | LSTM · 3 Earlier Boxes (synthetic) | weighted Jaccard: 0.025 -> 0.108 | Helped |
| Add input-cell attention over current and previous inputs | LSTM · 3 Middle Boxes (synthetic) | weighted Jaccard: 0.075 -> 0.106 | Helped |
| Apply input-cell attention only at the last 10 time steps rather than at each step (ablation) | LSTM+input-cell attention · Middle Box (synthetic) | saliency preservation over time | Hurt |
| Use input-cell attention to identify salient features | MNIST 3-class time-series classification · MNIST digits 1, 6, and 7 | accuracy drop after removing top salient features: 2% -> 9.5% | Helped |
| Use input-cell-attention salient features to remove features from the standard LSTM | Standard LSTM · MNIST 3-class time-series classification | accuracy drop after removing salient features: 6% | Helped |
| Use input-cell attention to identify salient features | LSTM · HCP on-task/off-task fMRI | top salient features from off-task period: more than 70% | Helped |
Closest papers by meaning
- Benchmarking Deep Learning Interpretability in Time Series Predictions (NeurIPS 2020 · similarity 0.88)
- Understanding and Visualizing Deep Visual Saliency Models (CVPR 2019 · similarity 0.87)
- What Do Deep Saliency Models Learn about Visual Attention? (NeurIPS 2023 · similarity 0.86)
- Self-Attention-Based Contextual Modulation Improves Neural System Identification (ICLR 2025 · similarity 0.85)
- Recurrent neural network dynamical systems for biological vision (NeurIPS 2024 · similarity 0.84)
- Great Minds Think Alike: The Universal Convergence Trend of Input Salience (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.