Research map / Architectures and efficiency / Attention and transformers
Enhancing Vision Transformer: Amplifying Non-Linearity in Feedforward Network Module
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes an improved feedforward network (IFFN) module for vision transformers that uses arbitrary GeLU (AGeLU) units and a depthwise spatial enhancement to reduce hidden dimensions, cutting FLOPs and parameters by roughly 10-21% without loss of ImageNet accuracy.
Problem
The FFN module in vision transformers is rarely modified compared with self-attention, and the reason changes to it help is unclear; the goal is to reduce FFN computation without hurting classification accuracy.
Approach
Shows the FFN acts as a non-linearity generator whose degree equals the hidden dimension, then uses learnable-affine AGeLU functions concatenated in two halves (halving hidden width) plus a depthwise conv with BN and GeLU to restore non-linearity via spatial information.
Key results
- On DeiT-Ti, IFFN reduces parameters by 12.6% and FLOPs by 12.7% while top-1 rises from 72.2% to 72.6% on ImageNet-1k.
- On PoolFormer models, IFFN cuts FLOPs by over 18% and parameters by over 17% with accuracy maintained or improved (e.g., PoolFormer-S24 80.3% to 80.7%).
- Directly replacing FFN with 4C separate nonlinear functions drops DeiT-Ti accuracy to 50.4% (GeLU) and 53.3% (AGeLU), versus 72.2% for the original FFN.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace the original FFN with IFFN | DeiT, Swin, PoolFormer, and LVT models · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 72.6; 79.9 -> 80.0; 81.8 -> 81.8; 81.2 -> 81.5; 83.2 -> 83.2; 83.5 -> 83.4; 77.2 -> 77.2; 80.3 -> 80.7; 81.4 -> | Mixed |
| use channel-wise enhancement only instead of the full IFFN (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 70.5 | Hurt |
| use spatial-wise enhancement only instead of the original FFN (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 72.8 | Helped |
| combine channel-wise and spatial-wise enhancement (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 72.6 | Helped |
| replace GeLU with SoftPlus within IFFN (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 71.6 -> 72.0 | Helped |
| replace GeLU with ELU within IFFN (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 71.1 -> 71.7 | Helped |
| replace GeLU with Swish within IFFN (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.0 -> 72.5 | Helped |
| increase the number of concatenated nonlinear functions from 2 to 4 (ablation) | DeiT-Ti IFFN · ImageNet-1k | Top-1 accuracy (%): 72.6 -> 68.6 | Hurt |
| replace the FFN with a direct sum of 4C GeLU functions (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 50.4 | Hurt |
| replace the FFN with a direct sum of 4C AGeLU functions (ablation) | DeiT-Ti · ImageNet-1k | Top-1 accuracy (%): 72.2 -> 53.3 | Hurt |
| increase depthwise convolution kernel size from 1 to 5 (ablation) | DeiT-Ti IFFN · ImageNet-1k | Top-1 accuracy (%): 72.0 -> 72.8 | Helped |
Surprising findings
- Naively replacing the FFN with 4C parallel nonlinear functions severely degrades accuracy (to about 50-53%), showing that the input-dependent bias is hard to emulate this way.
- Using four concatenated nonlinear functions instead of two reduces accuracy from 72.6% to 68.6% on DeiT-Ti.
Limitations
- Channel-wise enhancement alone degrades accuracy (DeiT-Ti 70.5%), so the method relies on the added spatial depthwise block to recover performance.
- Lipschitz analysis is only a bound given via an SDP formulation and is not empirically validated in the main text.
- Hardware and training compute are not reported.
Experimental setup
Datasets: ImageNet-1k COCO 2017 ADE20k
Benchmarks: ImageNet-1k COCO 2017 ADE20k
Metrics: top-1 accuracy FLOPs parameters Lipschitz bound
Baselines: DeiT Swin PoolFormer LVT SoftPlus ELU Swish
Builds on: GeLU ReActNet RPReLU VanillaNet series activation CMT inverted residual FFN Lipschitz SDP bound (Fazlyab et al.)
Closest papers by meaning
- Global Filter Networks for Image Classification (NeurIPS 2021 · similarity 0.86)
- Vision Transformer With Deformable Attention (CVPR 2022 · similarity 0.85)
- Equivariant Neural Functional Networks for Transformers (ICLR 2025 · similarity 0.85)
- Lite Vision Transformer With Enhanced Self-Attention (CVPR 2022 · similarity 0.85)
- Skip-Attention: Improving Vision Transformers by Paying Less Attention (ICLR 2024 · similarity 0.84)
- From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport (CVPR 2025 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.