Research map / Detection and segmentation / CNN features
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Introduces Gompertz Linear Unit (GoLU), a self-gated activation x·Gompertz(x) with a right-skewed Gumbel-CDF gate, and reports that it outperforms ReLU, GELU, Swish, Mish and others across vision, language, detection, segmentation and diffusion benchmarks.
Problem
Activation functions strongly influence deep network training dynamics, and popular choices (ReLU, GELU, Swish, Mish) have limitations such as dying neurons or high activation variance; the paper asks whether a different gate shape can improve learning across tasks.
Approach
Use the Gompertz function (CDF of standard Gumbel distribution) as the gate of a self-gated activation, GoLU(x)=x·exp(-exp(-x)). Its right-skewed asymmetry gives smaller gate values and slope near the origin, which reduces output variance compared with symmetric gates like Sigmoid (Swish) and Gaussian CDF (GELU).
Key results
- GoLU achieves top-1 accuracy of 76.63±0.03 on ResNet-50 and 80.72±0.04 on ViT-B/16 on ImageNet-1k, ranking first across most architectures versus GELU as second best.
- On GPT2-S (OpenWebText), GoLU reaches perplexity 17.297 vs 17.525 for GELU; on Mask R-CNN at lr=0.03 GoLU gets 39.36 box mAP vs 39.12 for GELU.
- Critical difference analysis across ImageNet-1k, MS-COCO, OWT, TS and CelebA ranks GoLU highest, followed by GELU.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace GELU with GoLU | ResNet-50 · ImageNet-1k | top-1 test accuracy: 76.07±0.06 -> 76.63±0.03 | Helped |
| replace Swish with GoLU | RN-20 · CIFAR-10 | top-1 test accuracy: 91.64±0.1 -> 91.77±0.1 | Helped |
| replace GELU with GoLU | babyGPT · TinyStories | perplexity: 4.462±0.005 -> 4.444±0.005 | Helped |
| replace GELU with GoLU | GPT2-S · OpenWebText | token accuracy: 44.262±0.042 -> 44.413±0.023 | Helped |
| replace ReLU with GoLU | DeepLabV3 ResNet-50 · MS-COCO, LR=0.02 | test mIoU: 64.99±0.173 -> 64.98±0.129 | No clear effect |
| use LR=0.01 instead of LR=0.02 (ablation) | DeepLabV3 ResNet-50 with GoLU · MS-COCO | test mIoU: 64.98±0.129 -> 65.98±0.124 | Helped |
| replace GELU with GoLU | Faster R-CNN-FPN ResNet-50 · MS-COCO | Box mAP: 38.16±0.044 -> 38.31±0.058 | Helped |
| replace GELU with GoLU | Mask R-CNN-FPN ResNet-50 · MS-COCO, LR=0.02 | Mask mAP: 34.73±0.024 -> 34.54±0.083 | Hurt |
| increase learning rate from 0.02 to 0.03 (ablation) | Mask R-CNN-FPN ResNet-50 with GoLU · MS-COCO | Box mAP: 38.96±0.101 -> 39.36±0.192 | Helped |
| replace Swish with GoLU | DDPM · CelebA, LR=0.0003 | test loss: 0.0196364±0.0 -> 0.0196419±0.0 | Hurt |
| increase learning rate from 0.0003 to 0.001 (ablation) | DDPM with GoLU · CelebA | test loss: 0.0196419±0.0 -> 0.0189506±0.0 | Helped |
| replace Swish with GoLU | Transformer-Big · WMT14 English–German | BLEU4: 28.34±0.08 -> 28.44±0.15 | Helped |
Surprising findings
- On Mask R-CNN at the default learning rate, GELU beats GoLU, and GoLU's advantage only appears after learning-rate tuning.
- The authors report GoLU as a second-to-last (negative) result on LC-PFN learning curve extrapolation, showing the gains do not transfer to every setup.
Limitations
- Several reported gains are small (often under one standard error-scale difference), and some tasks show GoLU ranking second or statistically insignificant differences.
- Effects attributed to variance reduction and weight spread are described as general trends not guaranteed to hold universally.
- Results at default learning rates are sometimes worse than baselines; gains depend on learning-rate tuning for some tasks.
Experimental setup
Datasets: ImageNet-1k CIFAR-10 TinyStories OpenWebText MS-COCO CelebA WMT14 English-German
Benchmarks: ImageNet-1k CIFAR-10 TinyStories OpenWebText MS-COCO CelebA WMT14 English-German
Metrics: top-1 accuracy perplexity token accuracy test loss mIoU box mAP mask mAP BLEU4 log-score
Baselines: ReLU LeakyReLU ELU GELU Swish Mish
Builds on: GELU Swish Mish
Closest papers by meaning
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.83)
- ReLUs Are Sufficient for Learning Implicit Neural Representations (ICML 2024 · similarity 0.83)
- Rethinking Neural Nonlinearity as Gating (NeurIPS 2026 · similarity 0.83)
- Neural Characteristic Activation Analysis and Geometric Parameterization for ReLU Networks (NeurIPS 2024 · similarity 0.83)
- Compelling ReLU Networks to Exhibit Exponentially Many Linear Regions at Initialization and During Training (ICML 2025 · similarity 0.83)
- Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning (NeurIPS 2026 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.