Research map / Learning theory and algorithms / Prediction and decision losses
Neural Networks Fail to Learn Periodic Functions and How to Fix It
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use Snake instead of ReLU, tanh, or standard periodic activations | Feedforward network regressing a periodic function · 1D periodic function extrapolation | periodic extrapolation quality | Helped |
| Use Snake instead of ReLU or Swish | Feedforward network · Wilshire 5000 test set | MSE: 0.0113^{ \pm 0.0002} -> 0.0089 ^{\pm 0.0002}; 0.0161^{ \pm 0.0007} -> 0.0089 ^{\pm 0.0002} | Helped |
| Use Snake instead of ARIMA(2,1,1) | Wilshire 5000 prediction · Wilshire 5000 test set | MSE: 0.0215^{ \pm 0.0075} -> 0.0089 ^{\pm 0.0002} | Helped |
| Replace ReLU with Snake | ResNet-18 · CIFAR-10 | testing accuracy: 93.5 \pm1.0\% -> 93.5 \pm1.0\% | No clear effect |
| Replace ReLU or Leaky-ReLU with Snake | ResNet-101 · CIFAR-10 | accuracy: 94.1\% | No clear effect |
| Use Snake instead of tanh or ReLU | Feedforward network · Minamitorishima weekly temperature prediction | extrapolation quality | Helped |
| Use Snake instead of tanh or ReLU | Feedforward network · Human body temperature prediction | periodic extrapolation quality | Helped |
| Use Snake instead of tanh, ReLU, sin, or sin + cos | Feedforward network · EUR-USD exchange-rate prediction | prediction quality | Helped |
| Use Snake instead of RNN | Periodic time-series predictor · Noisy sin(0.1x), training length T=100; validation t=101 to 300 | prediction performance | Helped |
| Use Snake instead of RNN | Periodic time-series predictor · Gradient update computation time | computation time: more than 10 times | Helped |
| Use variance correction by dividing post-activation values by sigma_a (ablation) | Snake ResNet-101 · CIFAR-10 | training speed and converged accuracy | Helped |
| Increase Snake frequency parameter a (ablation) | Snake periodic-function regressor · Rectangular and compound periodic functions | extrapolation quality | Helped |
Closest papers by meaning
- Neural Functions for Learning Periodic Signal (ICLR 2025 · similarity 0.89)
- Periodic Activation Functions Induce Stationarity (NeurIPS 2021 · similarity 0.88)
- Complexity of Neural Network Training and ETR: Extensions with Effectively Continuous Functions (AAAI 2024 · similarity 0.83)
- Towards Understanding the Universality of Transformers for Next-Token Prediction (ICLR 2025 · similarity 0.82)
- Rethinking Fourier Transform from A Basis Functions Perspective for Long-term Time Series Forecasting (NeurIPS 2024 · similarity 0.82)
- Global Convergence in Neural ODEs: Impact of Activation Functions (ICLR 2025 · similarity 0.82)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.