Research map / Architectures and efficiency / RNNs and normalization
Rewrite the Stars
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Explains why element-wise multiplication ("star operation") works by showing it implicitly maps inputs to a high-dimensional polynomial-kernel-like feature space, and proposes StarNet, a minimalist efficient CNN built on it.
Problem
The foundational rationale for why element-wise multiplication (star operation) outperforms summation in network design is unexplained; prior work offered only intuitive explanations.
Approach
Rewriting the star operation as a weighted sum of pairwise products of input channels shows it yields about (d+2)(d+1)/2 implicit dimensions in a d-dimensional space, akin to polynomial kernels; stacking l layers gives roughly (d/√2)^(2^l) dimensions. A minimalist network (StarNet) built on this operation is then proposed for efficient vision models.
Key results
- StarNet-S1 reaches 73.5% top-1 on ImageNet-1k at 0.7 ms iPhone13 latency, +2.1% over MobileOne-S0 at the same latency.
- StarNet-S4 reaches 78.4% top-1, 0.9% above EdgeViT-XS (77.5%) while about 3x faster on iPhone13 (1.0 ms vs 3.5 ms).
- In DemoNet, star beats sum at every width tested (e.g., 57.6% vs 51.1% at width 96), and removing all activations drops sum to 32.4% but star only to 70.5%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace summation with star operations | DemoNet, width 192 and depth 12 · ImageNet-1k | Top-1 accuracy: 66.2 -> 71.7 | Helped |
| replace summation with star operations | DemoNet, varying width · ImageNet-1k | Top-1 accuracy: 6.5↑; 5.5↑; 5.5↑; 5.5↑; 4.3↑; 3.9↑; 3.6↑ | Helped |
| replace summation with star operations | DemoNet, varying depth · ImageNet-1k | Top-1 accuracy: 6.5↑; 5.5↑; 4.7↑; 4.1↑; 4.0↑; 5.8↑; 4.8↑ | Helped |
| replace summation with star operations (ablation) | StarNet-S4 · ImageNet-1k | Top-1 accuracy: 75.3 -> 78.4 | Helped |
| add star operations in stages 2–4 (ablation) | StarNet-S4 with star only in stage 1 · ImageNet-1k | Top-1 accuracy: 75.1 -> 78.4 | Helped |
| add star operations in stages 3–4 (ablation) | StarNet-S4 with star in stages 1–2 · ImageNet-1k | Top-1 accuracy: 75.2 -> 78.4 | Helped |
| add star operation in stage 4 (ablation) | StarNet-S4 with star in stages 1–3 · ImageNet-1k | Top-1 accuracy: 76.8 -> 78.4 | Helped |
| add star operations in stages 1–2 (ablation) | StarNet-S4 with star in stages 3–4 · ImageNet-1k | Top-1 accuracy: 76.4 -> 78.4 | Helped |
| add star operations in stages 1–3 (ablation) | StarNet-S4 with star only in stage 4 · ImageNet-1k | Top-1 accuracy: 76.9 -> 78.4 | Helped |
| remove all activations (ablation) | DemoNet with star operation · ImageNet-1k | Top-1 accuracy: 71.7 -> 70.5 | Hurt |
| remove all activations (ablation) | DemoNet with summation · ImageNet-1k | Top-1 accuracy: 66.2 -> 32.4 | Hurt |
| remove activations (ablation) | StarNet-S4 · ImageNet-1k | Top-1 accuracy: 78.4 -> 75.6 | Hurt |
Surprising findings
- Removing all activations from DemoNet collapses summation to 32.4% top-1 while star retains 70.5%.
- Star operation adds essentially no latency on GPU and iPhone, though it is slower on CPU (9.4 ms vs 8.4 ms for StarNet-S4).
Limitations
- Coefficients of implicit high-dimensional dimensions are fixed/limited in tuning, which may explain only moderate gains from very high dimensions (acknowledged by authors).
- StarNet is a proof-of-concept with minimal engineering; not tuned for peak accuracy.
- Analysis is for a one-output, single-element simplification; the theoretical dimension count is approximate.
Experimental setup
Datasets: ImageNet-1k 2D moon dataset
Benchmarks: ImageNet-1k
Metrics: top-1 accuracy latency (ms) on iPhone13, P100 GPU, Intel E5-2680 CPU params FLOPs
Baselines: MobileNetv2 MobileNetv3 MobileOne ShuffleNetv2 GhostNet EdgeViT FasterNet EfficientNet-B0
Builds on: FocalNet HorNet VAN MobileNeXt DeiT
Closest papers by meaning
- Designing Network Design Spaces (CVPR 2020 · similarity 0.85)
- STAR: Synthesis of Tailored Architectures (ICLR 2025 · similarity 0.83)
- Do Deep Neural Network Solutions Form a Star Domain? (ICLR 2025 · similarity 0.81)
- From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awareness (ICLR 2022 · similarity 0.81)
- What Expressivity Theory Misses: Message Passing Complexity for GNNs (NeurIPS 2025 · similarity 0.81)
- PANDA: Expanded Width-Aware Message Passing Beyond Rewiring (ICML 2024 · similarity 0.80)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.