Research map / Architectures and efficiency / RNNs and normalization
EUGens: Efficient, Unified and General Dense Layers
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes EUGens, random-feature dense layers that disentangle weights and inputs, unbiasedly approximate polynomial-activation FFLs, and can replace FFLs in GPT-2, ViTs, NeRF and iSDF for faster inference.
Problem
Fully-connected feedforward layers create quadratic compute and parameter bottlenecks in Transformers and implicit neural representations, hindering real-time and resource-constrained deployment.
Approach
Generalize FFLs with random-feature maps applied separately to weights and inputs (with an input-norm term and higher-order Hadamard products), connected by a dot product, giving linear-time inference and unbiased approximation of polynomial-activation FFLs; also allows closed-form layer-wise distillation without backpropagation.
Key results
- EUGen-NeRF cuts inference time from 2.36s to 1.79s (about 24%) and storage from 3.27MB to 2.27MB with average PSNR 30.36 vs 30.45 for NeRF.
- EUGen-Mip-NeRF 360 is reported as 27% faster at inference with PSNR a few tenths of a dB lower than Mip-NeRF 360 across scenes.
- On ViT-Base, replacing six layers reduces inference parameters by nearly 30% while closely matching vanilla ViT accuracy on ImageNet and Places365.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Replace NeRF FFLs with EUGen layers | NeRF · Synthetic NeRF | inference time: 2.36s -> 1.79s | Helped |
| Replace NeRF FFLs with EUGen layers | NeRF · Synthetic NeRF | storage: 3.27MB -> 2.27MB | Helped |
| Replace NeRF FFLs with EUGen layers | NeRF · Synthetic NeRF | PSNR: 30.45 -> 30.36 | No clear effect |
| Use fixed rather than trainable projection matrices G (ablation) | EUGen-NeRF · Synthetic NeRF | PSNR: 30.36 -> 30.10 | Hurt |
| Use fixed rather than trainable projection matrices G (ablation) | EUGen-iSDF · iSDF | L1 norm: 4.17 -> 4.28 | Hurt |
| Replace iSDF MLP layers with EUGen layers | iSDF · iSDF | training time: 8.24 -> 7.81 | Helped |
| Replace the pretrained NeRF FFL using analytic layer-wise distillation | NeRF · NeRF lego | PSNR: 31.31 -> 28.54 | Hurt |
| Replace the pretrained NeRF FFL using EUGen layer-wise distillation | NeRF · NeRF lego | PSNR: 31.31 -> 31.25 | No clear effect |
| Replace the pretrained NeRF FFL using EUGen layer-wise distillation | NeRF · NeRF lego | inference time: 2.36 -> 1.84 | Helped |
| Use closed-form analytic distillation rather than the trainable EUGen distillation variant (ablation) | NeRF · NeRF lego | distillation time: 32.39 -> 0.12 | Helped |
| Increase the number of random features (ablation) | EUGen-iSDF · iSDF | Chamfer distance | Helped |
Surprising findings
- Pure polynomial-activation networks have limited expressiveness unless degree is large; the approximation results are argued to apply mainly in hybrid architectures with standard activations.
Limitations
- Speedups and accuracy shown on small-scale setups (GPT-2 124M, ViT-Base, NeRF variants); no large-scale LLM evaluation reported.
- Some EUGen variants show slightly worse quality (e.g., Mip-NeRF 360 PSNR and LPIPS drop; NeRF LPIPS higher).
Experimental setup
Datasets: OpenWebText ImageNet Places365 Synthetic NeRF Mip-NeRF 360 360_v2 ReplicaCAD
Benchmarks: ImageNet Places365 OpenWebText validation loss Synthetic NeRF Mip-NeRF 360 Zip-NeRF D-NeRF ReplicaCAD
Metrics: MSE validation loss validation accuracy PSNR SSIM LPIPS inference time Chamfer distance L1 norm
Baselines: SNNK Low-Rank layers GPT-2 ViT-Base NeRF Mip-NeRF 360 Zip-NeRF D-NeRF iSDF
Builds on: random features (Rahimi and Recht) SNNK Performer-style positive random features low-rank layers
Closest papers by meaning
- Scalable Neural Network Kernels (ICLR 2024 · similarity 0.86)
- Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.85)
- Sparse maximal update parameterization: A holistic approach to sparse training dynamics (NeurIPS 2024 · similarity 0.85)
- FFN Fusion: Rethinking Sequential Computation in Large Language Models (NeurIPS 2025 · similarity 0.84)
- Building on Efficient Foundations: Effective Training of LLMs with Structured Feedforward Layers (NeurIPS 2024 · similarity 0.84)
- Pay Attention to Small Weights (NeurIPS 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.