Research map / Architectures and efficiency / RNNs and normalization

Rethinking Neural Nonlinearity as Gating

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Theory of depth and width · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Proposes Threshold Gating (TG), a primitive that rewrites standard activations (ReLU, SiLU, GELU, sigmoid, tanh, Hardtanh) and softmax as input-conditioned soft gates over K affine branches, enabling lossless conversion of pretrained nets and training gains.

Problem

Activation functions are conventionally treated as discrete, post-MAC nonlinearities; this is costly for analog in-memory hardware where ADC/DAC crossings dominate power and area. The paper asks whether nonlinearity can instead be expressed as a single universal gating primitive.

Approach

Reframe every standard activation as a sum of sigmoid-gated affine branches applied before the MAC (threshold gating with K=2 or 3 branches), show exact or near-exact closed-form conversions, extend to softmax via a sigmoid-of-LSE identity, and prove a minimal branch theorem linking K to trainability.

Key results

• Converting pretrained ViT-B/16 GELU to K=3 TG recovers 0.00% accuracy change (vs -1.09% generic K=2); exact activations like ReLU, SiLU, sigmoid, tanh convert at 0.00% accuracy drop.
• Converted SmolLM-135M and TinyLlama-1.1B keep WikiText-2 perplexity unchanged (0.00%), while GPT-2 drifts +0.22% due to GELU approximation.
• TG_W K=2 training-step speedup of 1.47-2.04x over baselines across four model-dataset pairs (3 seeds each); DeepSpeech2 WER improves 1.2-4.5 points with TG_A on LSTM gates.

Surprising findings

• Softmax admits an exact two-branch sigmoid conversion via softmax(z)_i = sigmoid(z_i - LSE(z_{-i})) with no fine-tuning, so converted LLMs keep their perplexity.
• Multiplicative (input-carrying) branches on LSTM gates degrade WER back to baseline, while constant-family gating wins at every hidden size.

Limitations

• TG kernels are not hand-optimized; wall-clock training may be slower, and speedup is reported in steps rather than time.
• Circuit-level implementation is only a reference topology; crossbar integration and silicon validation are future work.
• GELU conversion is approximate (K=2 residual ~0.3%) and needs per-layer calibration or K=3 to be exact.

Experimental setup

Datasets: ImageNet CIFAR-10 CIFAR-100 MNIST AN4 WikiText-2 OpenWebText MSD Hippocampus

Benchmarks: ImageNet CIFAR-10 CIFAR-100 MNIST AN4 WikiText-2 OpenWebText MSD Hippocampus

Metrics: top-1 accuracy perplexity word error rate Dice training-step speedup parameter count

Baselines: ReLU SiLU GELU LSTM with sigmoid/tanh gates Maxout SwiGLU

Builds on: SiLU GELU Mixture of Experts Highway Networks Hayou et al. edge-of-chaos analysis

Closest papers by meaning

• Analog Foundation Models (NeurIPS 2025 · similarity 0.86)
• Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.85)
• DEEP NEURAL NETWORK INITIALIZATION WITH SPARSITY INDUCING ACTIVATIONS (ICLR 2024 · similarity 0.85)
• Towards Exact Gradient-based Training on Analog In-memory Computing (NeurIPS 2024 · similarity 0.85)
• Exploiting Symmetric Temporally Sparse BPTT for Efficient RNN Training (AAAI 2024 · similarity 0.84)
• Exact, Tractable Gauss-Newton Optimization in Deep Reversible Architectures Reveal Poor Generalization (NeurIPS 2024 · similarity 0.84)