Research map / Large language models / Language, safety and interpretability
Polynomial Composition Activations: Unleashing the Dynamics of Large Language Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes PolyCom, polynomial composition activations (PolyReLU, PolyNorm) for transformer FFNs, with approximation-theory results and LLM pretraining gains over SwiGLU/GELU/ReLU.
Problem
Standard transformer activations (ReLU, GELU, SwiGLU) are limited in modeling higher-order interactions, and prior pure polynomial activations are known to underperform or be unstable.
Approach
Define PolyCom as a learnable weighted sum of powers composed with a base function (Type I: sum a_i rho(x)^i, e.g. PolyReLU; Type II: sum a_i rho(x^i) with normalization, e.g. PolyNorm), used as the FFN activation; theoretically show PolyReLU networks achieve optimal Sobolev approximation rate O(eps^{-d/n}).
Key results
- On the 1B dense model after 250B tokens, PolyNorm reaches average downstream accuracy 58.68 vs 57.47 for SwiGLU, and validation PPL 3.17 vs 3.22.
- On MoE (OLMoE-1B-7B, 200B tokens), PolyNorm lowers average validation loss from 2.41 (SwiGLU) to 2.39 and improves average downstream accuracy from 62.78 to 63.37.
- Theory: PolyReLU networks achieve the optimal approximation rate O(eps^{-d/n}) on Sobolev unit balls, versus O(eps^{-d/n} ln(1/eps)) for ReLU networks.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace SwiGLU with PolyNorm | 1B dense model · 1B dense model, 250B tokens | average downstream accuracy: 57.47 -> 58.68 | Helped |
| Replace SwiGLU with PolyReLU | 1B dense model · 1B dense model, 250B tokens | validation perplexity: 3.22 -> 3.18 | Helped |
| Replace SwiGLU with PolyNorm | OLMoE-1B-7B · MoE downstream tasks | average accuracy: 62.78 -> 63.37 | Helped |
| Replace SwiGLU with PolyNorm | OLMoE-1B-7B · ARC-Easy | accuracy: 71.58 -> 70.70 | Hurt |
| Replace SwiGLU with PolyNorm | OLMoE-1B-7B · WinoGrande | accuracy: 62.75 -> 62.19 | Hurt |
| Increase PolyReLU order from 2 to 3 (ablation) | PolyReLU · 1B dense model training | convergence speed | Helped |
| Increase PolyReLU order from 3 to 4 (ablation) | PolyReLU · 1B dense model training | convergence speed | No clear effect |
| Use PolyNorm composition instead of PolyReLU composition (ablation) | PolyCom · 1B dense model training | training loss | Helped |
| Combine ReLU with normalization (PolyReLUNorm) instead of using PolyNorm (ablation) | PolyCom · 1B dense model training | training loss | Hurt |
| Use PolyReLU instead of ReLU (ablation) | 1B dense model · downstream tasks | accuracy | Helped |
| Use PolyReLU instead of ReLU^2 (ablation) | 1B dense model · downstream tasks | accuracy | Helped |
| Use PolyReLU or PolyNorm instead of conventional activations (ablation) | transformer FFN layers · 1B dense model | effective weight rank | Helped |
Surprising findings
- Pure ReLU and GELU baselines perform roughly on par, with ReLU slightly above GELU on average accuracy in the dense setting.
- Increasing polynomial order beyond 3 gave no noticeable convergence gain while adding overhead and overflow risk.
Limitations
- Gains on the MoE model are small (average validation loss improves by only 0.02; downstream avg +0.59).
- PolyReLU lacks normalization and may be unstable in non-transformer architectures such as ResNet.
- Theory compares against ReLU networks; GELU and SwiGLU are only handled via local Taylor approximation.
Experimental setup
Datasets: RedPajama-1T OLMoE Mix
Benchmarks: ARC-Easy ARC-Challenge HellaSwag PIQA SciQ Winogrande MMLU var BoolQ COPA OpenbookQA CoQA CSQA
Metrics: accuracy training loss validation perplexity validation loss
Baselines: SwiGLU GELU ReLU ReLU^2 square ReLU
Builds on: Llama 2 OLMoE SwiGLU Π-nets rectified power units
Closest papers by meaning
- PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels (ICML 2024 · similarity 0.87)
- Transformer Block Coupling and its Correlation with Generalization in LLMs (ICLR 2025 · similarity 0.87)
- Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models (NeurIPS 2024 · similarity 0.86)
- Do Efficient Transformers Really Save Computation? (ICML 2024 · similarity 0.86)
- Understanding Transformers via N-Gram Statistics (NeurIPS 2024 · similarity 0.86)
- Separations in the Representational Capabilities of Transformers and Recurrent Architectures (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.