Research map / Architectures and efficiency / Attention and transformers
ParameterNet: Parameters Are All You Need for Large-scale Visual Pretraining of Mobile Networks
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Identifies a "low FLOPs pitfall" where low-FLOPs networks don't benefit from ImageNet-22K pretraining, and proposes ParameterNet, which adds parameters via dynamic convolution at near-constant FLOPs, plus a LLaMA-1B MoE-style extension for language.
Problem
Low-FLOPs mobile vision networks fail to benefit from large-scale supervised pretraining (ImageNet-22K), as their accuracy gains do not follow the pretraining data scale.
Approach
Increase the number of parameters of low-FLOPs networks without increasing FLOPs by replacing standard convolutions with dynamic convolutions (input-conditioned mixture of M expert kernels), so that large-scale pretraining helps mobile-scale models.
Key results
- ParameterNet-600M pretrained on ImageNet-22K reaches 81.6% top-1 on ImageNet-1K at 0.6G FLOPs, versus 80.9% for Swin-T at 4.5G FLOPs.
- ImageNet-22K pretraining improves ParameterNet-300M from 77.2% to 79.1% and ParameterNet-600M from 79.0% to 81.6%, while GhostNet loses accuracy under the same pretraining.
- On LLaMA-1B, adding 8 experts on the up projection raises average zero-shot accuracy from 58.19 to 60.56 at unchanged FLOPs (919G).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pretrain GhostNet-300M on ImageNet-22K instead of ImageNet-1K | GhostNet-300M · ImageNet-1K val | Top-1 accuracy: 76.3 -> 75.7 (-0.6) | Hurt |
| Add dynamic convolution with 4 experts and pretrain on ImageNet-22K | GhostNet-300M · ImageNet-1K val | Top-1 accuracy: 77.2 -> 79.1 (+1.9) | Helped |
| Pretrain GhostNet-600M on ImageNet-22K instead of ImageNet-1K | GhostNet-600M · ImageNet-1K val | Top-1 accuracy: 78.6 -> 78.3 (-0.3) | Hurt |
| Add dynamic convolution with 4 experts and pretrain on ImageNet-22K | GhostNet-600M · ImageNet-1K val | Top-1 accuracy: 79.0 -> 81.6 (+2.6) | Helped |
| Add sparse-activated MoE experts to the LLaMA-1B up projection | LLaMA-1B · Average across language tasks | Accuracy: 58.19 -> 60.56 | Helped |
| Increase dynamic experts from 1 to 8 (ablation) | GhostNet-300M · ImageNet-1K val, ImageNet-1K pretraining | Top-1 accuracy: 76.3 -> 77.7 | Helped |
| Increase dynamic experts from 1 to 8 (ablation) | GhostNet-300M · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 75.7 -> 79.4 | Helped |
| Apply ParameterNet to Swin-300M (ablation) | Swin-300M · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 67.0 (-2.2) -> 74.5 (+2.2) | Helped |
| Apply ParameterNet to EfficientNetV2-B0 (ablation) | EfficientNetV2-B0 · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 77.6 (-1.1) -> 81.5 (+1.7) | Helped |
| Replace dynamic convolution with re-parameterized convolution (ablation) | GhostNet-300M · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 79.1 -> 76.9 | Hurt |
| Increase dynamic experts from 2 to 4 (ablation) | GhostNet-300M · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 77.7 -> 79.1 | Helped |
| Increase dynamic experts from 4 to 8 (ablation) | GhostNet-300M · ImageNet-1K val, ImageNet-22K pretraining | Top-1 accuracy: 79.1 -> 79.4 | Helped |
Surprising findings
- Standard GhostNet and EfficientNetV2 pretrained on ImageNet-22K perform worse than ImageNet-1K pretraining at low FLOPs (e.g., GhostNet-600M 78.3 vs 78.6), the 'low FLOPs pitfall'.
- Re-parameterized convolution, which adds training-time parameters only, does not improve ImageNet-22K pretrained accuracy (76.9 vs 76.8 on 1K), unlike dynamic convolution.
Limitations
- Experiments on the vision side are limited to ImageNet-22K and ImageNet-1K classification, with downstream detection and segmentation not evaluated in the main text.
- Language-domain validation uses a single small LLaMA-1B variant with zero-shot commonsense tasks and only a training-loss comparison.
- Compute and training hardware details are not reported, and code release is only promised as a future link.
Experimental setup
Datasets: ImageNet-22K ImageNet-1K C4 Wikipedia ArXiv
Benchmarks: ImageNet-1K ARC (easy) BoolQ SST-2 HellaSwag
Metrics: top-1 accuracy FLOPs parameters latency training loss zero-shot accuracy
Baselines: GhostNet Swin Transformer EfficientNetV2 ResNet50 (BiT) ResNet50 (Billion-scale) ResNet101 (JFT-300M) EfficientNet-B0 LLaMA-1B
Builds on: GhostNet dynamic convolution (CondConv/DyNet) Switch Transformer EfficientNetV2 Swin Transformer LLaMA
Closest papers by meaning
- ImageNet-21K Pretraining for the Masses (NeurIPS 2021 · similarity 0.86)
- LSNet: See Large, Focus Small (CVPR 2025 · similarity 0.86)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.86)
- Can We Scale Transformers to Predict Parameters of Diverse ImageNet Models? (ICML 2023 · similarity 0.86)
- Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective (AAAI 2025 · similarity 0.85)
- Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit (ICLR 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.