AI Paper Atlas

Research map / Detection and segmentation / CNN features

Deep Incubation: Training Large Models by Divide-and-Conquering

Zanlin Ni, Yulin Wang, Jiangwei Yu, Haojun Jiang, Yue Cao, Gao Huang

ICCV 2023 · Detection and segmentation › CNN features · approach: Efficient DETR query design · 12 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
train sub-modules independently with a shared meta model then assemble and fine-tuneViT-H · ImageNet-1Ktop-1 accuracy: +2.7%Improved
use Deep Incubation instead of E2E training for training efficiencyViT-L and ViT-H · ImageNet-1Ktraining cost at similar accuracy: 3.2x and 4.1x less costImproved
replace modular training with E2E training before the same fine-tuning stage (ablation)DeiT-T-128 · CIFAR-100top-1 accuracy: 67.9 -> 77.2Improved
pre-train the meta model (ablation)Deep Incubation · CIFAR-100top-1 accuracy: 70.9 -> 76.4Improved
fix the meta model during incubation (ablation)Deep Incubation · CIFAR-100top-1 accuracy: 76.4 -> 77.2Improved
use module incubation rather than module imitation (ablation)Deep Incubation · CIFAR-100top-1 accuracyImproved
use shallower meta models in module incubation (ablation)Deep Incubation · CIFAR-100top-1 accuracyImproved
fine-tune assembled model for only one epoch (ablation)Deep Incubation · CIFAR-100top-1 accuracy: 69.4 -> 71.2Improved
increase number of modules K from 2 to 4 for DeiT-T-32 (ablation)DeiT-T-32 · CIFAR-100top-1 accuracy: 72.3 -> 76.1; K=16 gives 75.6Mixed
use Deep Incubation with less training dataViT-B · ImageNet-1K 25% subsettop-1 accuracy: 65.7 -> 72.9Improved
make ViT-B deeper and narrower at the same FLOPsE2E-DeiT · ImageNet-1Ktop-1 accuracy: 81.8 -> 78.7Hurt
make ViT-B deeper and narrower at the same FLOPsDeep Incubation · ImageNet-1Ktop-1 accuracy: 82.4 -> 82.7Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy