Research map / Learning theory and algorithms / Kernels and regression theory
Avoiding Kernel Fixed Points: Computing with ELU and GELU Infinite Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use GELU instead of ReLU as the GP kernel activation | Single-hidden-layer GP regression · Boston, Concrete, Energy, Wine, Yacht | RMSE: Boston: 2.85 ± 0.64 -> 2.86 ± 0.65; Concrete: 5.22 ± 0.55 -> 5.21 ± 0.56; Energy: 0.89 ± 0.11 -> 0.92 ± 0.12; Wine: 1.15 | No clear effect |
| Use dataset-specific activation kernels rather than one kernel for all datasets | Single-hidden-layer GP regression · Bayesian deep learning benchmarks | RMSE | Mixed |
| Use GELU instead of ReLU with increasing depth | Deep GP regression on synthetic regression data · Synthetic regression, depth 1 to 100 | Training and test MSE | Mixed |
| Use activations without a unique kernel fixed point (GELU or ELU) rather than ReLU | Deep infinitely wide MLP kernels · Inputs on a constant-norm ball | Kernel fixed-point uniqueness | Helped |
Closest papers by meaning
- An Infinite-Width Analysis on the Jacobian-Regularised Training of a Neural Network (ICML 2024 · similarity 0.88)
- Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $\mu$ Parametrization (ICML 2025 · similarity 0.87)
- Deep Kernel Posterior Learning under Infinite Variance Prior Weights (ICLR 2025 · similarity 0.87)
- Convolutional Deep Kernel Machines (ICLR 2024 · similarity 0.87)
- Adaptive kernel predictors from feature-learning infinite limits of neural networks (ICML 2025 · similarity 0.87)
- Three Mechanisms of Feature Learning in a Linear Network (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.