Research map / Architectures and efficiency / RNNs and normalization
Multilinear Operator Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes MONet, a polynomial network built solely from multilinear operations (Mu-Layer with Hadamard products and no activation functions) that approaches modern vision architectures on ImageNet-1K and other benchmarks.
Problem
Polynomial networks avoid activation functions and enable privacy-preserving (homomorphic encryption) and interpretable models, but they have lagged far behind modern architectures on image recognition benchmarks.
Approach
A Mu-Layer computes (AX) * (BDX) plus a shortcut and linear term, a two-branch Hadamard-product block with low-rank factorization; stacking Poly-Blocks of two Mu-Layers with token-based (MLP-style) inputs and pyramid patch embeddings yields high-degree (up to 4 per block) interactions without activations.
Key results
- Multi-stage MONet-S reaches 81.3% top-1 on ImageNet-1K with 32.9M params and 6.8B FLOPs, versus 70.2% for the best prior polynomial net (R-PolyNets) reported in the table.
- Multi-stage MONet-T reaches 77.0% top-1 on ImageNet-1K with 10.3M params, outperforming prior polynomial networks by a wide margin.
- On ImageNet-100 ablation, the Poly-Block (two Mu-Layers) gets 82.94% top-1 versus 55.11% for a linear block with the same depth and width.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use MONet-S instead of HireMLP | ImageNet-C classification · ImageNet-C | mCE: 51.9 -> 49.7 | Helped |
| Use MONet-S instead of CycleMLP-T | ImageNet1K classification · ImageNet1K val | Top-1 accuracy: 81.3 -> 81.3 | No clear effect |
| Use MONet-T instead of D-PolyNets | Image classification · Tiny ImageNet | Accuracy: 61.8 -> 61.5 | Hurt |
| Remove spatial shift from the first Mu-Layer (ablation) | MONet Poly-Block · ImageNet100 | Top-1 accuracy: 82.94\% -> 81.50\% | Hurt |
| Replace the second Mu-Layer with an MLP layer (ablation) | MONet Poly-Block · ImageNet100 | Top-1 accuracy: 82.94\% -> 67.61\% | Hurt |
| Replace both Mu-Layers with MLP layers (ablation) | MONet Poly-Block · ImageNet100 | Top-1 accuracy: 82.94\% -> 55.11\% | Hurt |
| Use pyramid patch embedding with patch sizes 14 and 7 instead of single-level patch size 14 (ablation) | MONet · ImageNet100 | Top-1 accuracy: 79.54 -> 82.94 | Helped |
| Increase hidden size from 96 to 384 (ablation) | MONet · ImageNet100 | Top-1 accuracy: 67.5\% -> 82.94\% | Helped |
| Increase depth from 1 to 16 (ablation) | MONet · ImageNet100 | Top-1 accuracy: 44.25 -> 82.94 | Helped |
| Set shrinkage ratio to 2 instead of 4 (ablation) | MONet · ImageNet100 | Top-1 accuracy: 82.94 -> 83.88 | Helped |
| Increase shrinkage ratio from 4 to 8 (ablation) | MONet · ImageNet100 | Top-1 accuracy: 82.94 -> 82.28 | Hurt |
Surprising findings
- Replacing the second Mu-Layer's spatial shift and using two Mu-Layers without spatial shift costs only about 1.4 points (81.50% vs 82.94%), indicating the spatial shift is not crucial.
- The paper reports that MLP-Mixer's accuracy drops sharply when hidden size falls from 128 to 96 (70.05% to 48.3%), while MONet degrades more gracefully.
Limitations
- No theoretical characterization of which polynomial expansions MONet can express.
- Performance still trails the strongest modern models on some benchmarks; on ImageNet-100 ablations it is not compared with full-scale models.
- Depth N>10 in practice; degree of expansion needed for a given task is not theoretically established.
Experimental setup
Datasets: ImageNet-1K ImageNet-100 CIFAR10 SVHN Oxford Flower102 Tiny ImageNet ImageNet-C MedMNIST ADE20K Lotka-Volterra simulated data
Benchmarks: ImageNet-1K CIFAR10 SVHN Oxford Flower102 Tiny ImageNet ImageNet-C MedMNIST ADE20K
Metrics: top-1 accuracy top-5 accuracy mCE mIoU FLOPs parameters
Baselines: Pi-Nets Hybrid Pi-Nets PDC R-PolyNets D-PolyNets MLP-Mixer ResMLP S2MLP CycleMLP ResNet-50 DeiT-S ViT-B/16
Builds on: Pi-Nets MLP-Mixer CycleMLP Neural ODE Augmented Neural ODE
Closest papers by meaning
- Regularization of Polynomial Networks for Image Recognition (CVPR 2023 · similarity 0.85)
- P-nets: Deep Polynomial Neural Networks (CVPR 2020 · similarity 0.85)
- Identifiability of Deep Polynomial Neural Networks (NeurIPS 2025 · similarity 0.85)
- Low-Rank Adaptation in Multilinear Operator Networks for Security-Preserving Incremental Learning (CVPR 2025 · similarity 0.84)
- MogaNet: Multi-order Gated Aggregation Network (ICLR 2024 · similarity 0.84)
- Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices (NeurIPS 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.