Research map / Architectures and efficiency / RNNs and normalization
Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes a weight initialization for tanh feedforward networks, W = D + Z with identity-like D and scaled Gaussian noise, motivated by fixed-point analysis of tanh(ax), to avoid activation vanishing and improve robustness and data efficiency over Xavier.
Problem
Xavier initialization for tanh networks causes activations to vanish with depth and yields performance that degrades with network size and depends on training data amount and random seed, especially in PINNs.
Approach
Uses a fixed-point analysis of tanh(ax) to argue that a per-layer gain a near 1 keeps activations from vanishing; the initial weight matrix is an identity-like (mod-wrapped) matrix plus Gaussian noise with std alpha/sqrt(fan_in), alpha=0.085, so the effective gain is distributed around 1.
Key results
- On 10-layer FFNNs with 64 nodes trained on 10 samples of MNIST, the proposed initialization reaches 37.32% accuracy vs 31.13% for Xavier (Table of data efficiency).
- On 20-hidden-layer FFNNs, the proposed method is best in most width settings; e.g., MNIST at 2 nodes: 62.82% vs 49.78% for Xavier.
- In PINN experiments with 16 nodes, the proposed method attains the lowest median loss at nearly all depths across Allen-Cahn, Burgers, Diffusion, and Poisson equations (e.g., Diffusion 5 layers: 9.14e-04 vs 2.52e-03 for Xavier).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use proposed tanh weight initialization instead of Xavier | tanh FFNN · MNIST, 20 layers, 512 nodes | validation accuracy: 97.35 -> 97.96 | Helped |
| Use proposed tanh weight initialization instead of Xavier | tanh FFNN · FMNIST, 20 layers, 512 nodes | validation accuracy: 88.28 -> 88.59 | Helped |
| Use proposed tanh weight initialization instead of Xavier | tanh FFNN · CIFAR-10, 20 layers, 512 nodes | validation accuracy: 51.71 -> 52.21 | Helped |
| Use proposed tanh weight initialization instead of Xavier | tanh FFNN · CIFAR-100, 20 layers, 32 nodes | validation accuracy: 23.71 -> 24.93 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 64-node tanh FFNN · MNIST, 100 layers | validation accuracy: 94.08 -> 96.06 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 64-node tanh FFNN · FMNIST, 100 layers | validation accuracy: 83.41 -> 86.01 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 64-node tanh FFNN · CIFAR-10, 100 layers | validation accuracy: 46.71 -> 48.96 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 64-node tanh FFNN · CIFAR-100, 100 layers | validation accuracy: 20.32 -> 24.41 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 10-layer tanh FFNN · MNIST, 10 samples | validation accuracy: 31.13 -> 37.32 | Helped |
| Use proposed tanh weight initialization instead of Xavier | 10-layer tanh FFNN · FMNIST, 10 samples | validation accuracy: 36.16 -> 37.31 | Helped |
| Use proposed tanh weight initialization instead of Xavier | tanh PINN · Allen-Cahn, 16 nodes, 50 layers | PINN loss: 1.24e-03 -> 4.03e-04 | Helped |
| Use proposed tanh weight initialization instead of Xavier | tanh PINN · Burgers, 16 nodes, 60 layers | PINN loss: 6.5e-02 -> 3.43e-04 | Helped |
Surprising findings
- Xavier with batch or layer normalization often performs worse than plain Xavier on feature-rich datasets such as CIFAR-10 and CIFAR-100.
- Xavier+LN diverges badly in several PINN configurations (e.g., losses of 1e2 to 1e3 at deeper depths).
Limitations
- Theoretical analysis covers a simplified setting with equal layer widths; the noise scale alpha=0.085 was chosen experimentally.
- Experiments use a fixed learning rate of 0.0001 or 0.001 without learning-rate decay and short training budgets (20-100 epochs for classification).
- Reported results are medians/averages over 5-10 runs with no significance tests shown.
Experimental setup
Datasets: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 Allen-Cahn Burgers Diffusion Poisson
Benchmarks: MNIST Fashion-MNIST CIFAR-10 CIFAR-100 Allen-Cahn Burgers Diffusion Poisson
Metrics: validation accuracy validation loss PINN loss mean absolute error
Baselines: Xavier initialization Xavier + batch normalization Xavier + layer normalization He initialization + BN orthogonal initialization
Builds on: Xavier initialization physics-informed neural networks Adam L-BFGS
Closest papers by meaning
- Principled Weight Initialisation for Input-Convex Neural Networks (NeurIPS 2023 · similarity 0.86)
- Sinusoidal Initialization, Time for a New Start (NeurIPS 2025 · similarity 0.85)
- Initializing Models with Larger Ones (ICLR 2024 · similarity 0.85)
- Fast Training of Sinusoidal Neural Fields via Scaling Initialization (ICLR 2025 · similarity 0.84)
- On the Impacts of the Random Initialization in the Neural Tangent Kernel Theory (NeurIPS 2024 · similarity 0.84)
- The Effect of Weight Precision on the Neuron Count in Deep ReLU Networks (ICML 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.