Research map / Architectures and efficiency / RNNs and normalization
Efficient Activation Function Optimization through Surrogate Modeling
Garrett Bingham, Risto Miikkulainen
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace ReLU with AQuaSurF-discovered HardSigmoid(HardSigmoid(x))*ELU(x) | All-CNN-C · CIFAR-100 | accuracy: 0.6897 -> 0.6990 | Improved |
| replace ReLU with discovered Swish(-2x) | ResNet-56 · CIFAR-100 | accuracy: 0.7348 -> 0.7469 | Improved |
| replace ReLU with discovered -x*sigma(x)*HardSigmoid(x) | MobileViTv2-0.5 · ImageNet | accuracy: 0.6139 -> 0.6396 | Improved |
| transfer discovered -x*sigma(x)*HardSigmoid(x) to ResNet-50 instead of ReLU | ResNet-50 · ImageNet | top-1 accuracy: 0.7660 -> 0.7776 | Improved |
| transfer discovered x*erfc(ELU(x)) to ResNet-50 instead of ReLU | ResNet-50 · ImageNet | top-1 accuracy: 0.7660 -> 0.7635 | Hurt |
| replace ReLU with discovered erfc(Softplus(x))^2 (sigmoidal) | CoAtNet · Imagenette | accuracy: 0.8772 -> 0.8907 | Improved |
| use sigmoid instead of ReLU as activation | All-CNN-C · CIFAR-100 | accuracy: 0.6897 -> 0.0100 | Hurt |
| use Swish instead of ReLU | MobileViTv2-0.5 · ImageNet | accuracy: 0.6139 -> 0.6383 | Improved |
| extend search with ELiSH baseline instead of ReLU | CoAtNet · Imagenette | accuracy: 0.8772 -> 0.1000 | Hurt |
| use FIM eigenvalues plus function outputs instead of either feature alone for surrogate (ablation) | KNR/RFR/SVR search · Act-Bench datasets | best validation accuracy found | Improved |
| use UMAP-feature regression instead of random search (ablation) | KNR/RFR/SVR search · Act-Bench datasets | best validation accuracy found | Improved |
| use KNR instead of RFR or SVR as search algorithm (ablation) | AQuaSurF · Act-Bench datasets | best validation accuracy found | Improved |
Closest papers by meaning
- Bayesian Optimization for Sparse Neural Networks With Trainable Activation Functions (TPAMI 2024 · similarity 0.88)
- Entropy-based Activation Function Optimization: A Method on Searching Better Activation Functions (ICLR 2025 · similarity 0.88)
- A generalized neural tangent kernel for surrogate gradient learning (NeurIPS 2024 · similarity 0.85)
- Active Learning Guided by Efficient Surrogate Learners (AAAI 2024 · similarity 0.85)
- Better NTK Conditioning: A Free Lunch from (ReLU) Nonlinear Activation in Wide Neural Networks (NeurIPS 2025 · similarity 0.84)
- LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs (AAAI 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
