Research map / Optimization / Optimizers
Noisy Activation Functions
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace sigmoid and tanh with noisy hard-saturating activations | LSTM on Penn Treebank word-level · Penn Treebank word-level | test perplexity: 115.6 -> 108.0 | Helped |
| Replace sigmoid and tanh with noisy hard-saturating activations | NMT · Europarl newstest2011 | BLEU: 20.18 -> 22.57 | Helped |
| Use hard-saturating rather than soft-saturating activations | NMT · Europarl newstest2011 | BLEU: 20.18 -> 21.59 | Helped |
| Use noisy activations | Soft attention captioning model · Flickr8k | METEOR: 18.9 -> 20.5 | Helped |
| Use noisy activations | Soft attention captioning model · Flickr8k | BLEU-1: 67 -> 66 | Hurt |
| Add dropout to the noisy activation model (ablation) | Soft attention captioning model · Flickr8k | METEOR: 20.3 -> 20.5 | No clear effect |
| Use half-normal rather than normal noise (ablation) | Noisy LSTM · Penn Treebank word-level | test perplexity: 108.0 -> 108.7 | Hurt |
| Replace sigmoid and tanh with noisy activations | Reference model · Learning to Execute | test accuracy: 46.45% -> 48.09% | Helped |
| Anneal the noise scale during training (ablation) | Noisy LSTM+MLP (NAN) · Unique-element sequence task | test error: 31.12 -> 9.53 | Helped |
| Use NANIL rather than NAN with annealed noise (ablation) | Noisy LSTM+MLP · Unique-element sequence task | test error: 9.53 -> 20.94 | Hurt |
| Use normal input noise rather than output noise (ablation) | Character-level GRU language model · Penn Treebank | training performance | No clear effect |
| Add noise only when the activation saturates rather than throughout training inputs (ablation) | Noisy activation function · Character-level Penn Treebank | training progress | Mixed |
Closest papers by meaning
- Rethinking Neural Nonlinearity as Gating (NeurIPS 2026 · similarity 0.87)
- On the Complexity of Learning Neural Networks (NeurIPS 2017 · similarity 0.86)
- Spike-timing-dependent Hebbian learning as noisy gradient descent (NeurIPS 2025 · similarity 0.84)
- Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics (NeurIPS 2025 · similarity 0.84)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.84)
- DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.