Research map / Detection and segmentation / CNN features
Deep Incubation: Training Large Models by Divide-and-Conquering
Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, Gao Huang
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| train sub-modules independently with a shared meta model then assemble and fine-tune | ViT-H · ImageNet-1K | top-1 accuracy: +2.7% | Improved |
| use Deep Incubation instead of E2E training for training efficiency | ViT-L and ViT-H · ImageNet-1K | training cost at similar accuracy: 3.2x and 4.1x less cost | Improved |
| replace modular training with E2E training before the same fine-tuning stage (ablation) | DeiT-T-128 · CIFAR-100 | top-1 accuracy: 67.9 -> 77.2 | Improved |
| pre-train the meta model (ablation) | Deep Incubation · CIFAR-100 | top-1 accuracy: 70.9 -> 76.4 | Improved |
| fix the meta model during incubation (ablation) | Deep Incubation · CIFAR-100 | top-1 accuracy: 76.4 -> 77.2 | Improved |
| use module incubation rather than module imitation (ablation) | Deep Incubation · CIFAR-100 | top-1 accuracy | Improved |
| use shallower meta models in module incubation (ablation) | Deep Incubation · CIFAR-100 | top-1 accuracy | Improved |
| fine-tune assembled model for only one epoch (ablation) | Deep Incubation · CIFAR-100 | top-1 accuracy: 69.4 -> 71.2 | Improved |
| increase number of modules K from 2 to 4 for DeiT-T-32 (ablation) | DeiT-T-32 · CIFAR-100 | top-1 accuracy: 72.3 -> 76.1; K=16 gives 75.6 | Mixed |
| use Deep Incubation with less training data | ViT-B · ImageNet-1K 25% subset | top-1 accuracy: 65.7 -> 72.9 | Improved |
| make ViT-B deeper and narrower at the same FLOPs | E2E-DeiT · ImageNet-1K | top-1 accuracy: 81.8 -> 78.7 | Hurt |
| make ViT-B deeper and narrower at the same FLOPs | Deep Incubation · ImageNet-1K | top-1 accuracy: 82.4 -> 82.7 | Improved |
Closest papers by meaning
- MetaInit: Initializing learning by learning to initialize (NeurIPS 2019 · similarity 0.85)
- ImageNet-21K Pretraining for the Masses (NeurIPS 2021 · similarity 0.84)
- Large-batch Optimization for Dense Visual Predictions: Training Faster R-CNN in 4.2 Minutes (NeurIPS 2022 · similarity 0.84)
- DeepOBS: A Deep Learning Optimizer Benchmark Suite (ICLR 2019 · similarity 0.84)
- ZipIt! Merging Models from Different Tasks without Training (ICLR 2024 · similarity 0.84)
- Model LEGO: Creating Models Like Disassembling and Assembling Building Blocks (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
