Research map / Architectures and efficiency / Pruning and efficient tuning
Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models
Jiawei Fan, Shigeng Wang, Chao Li, Xiaolong Liu, Anbang Yao
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add inverse weight initialization (IWI) to CoM-PT at 24 epochs (ablation) | CoM-PT ViT-S · ImageNet-1K | top-1: 26.48 -> 28.03 | Improved |
| add inverse feature distillation (IFD) to CoM-PT at 24 epochs (ablation) | CoM-PT ViT-S · ImageNet-1K | top-1: 26.48 -> 29.29 | Improved |
| combine IWI and IFD (ablation) | CoM-PT ViT-S · ImageNet-1K | top-1: 28.03 -> 30.24 | Improved |
| combine IWI and IFD (ablation) | CoM-PT ViT-S · COCO retrieval | R@1: 30.44 -> 34.15 | Improved |
| use the model chain (initialize from predecessor with distillation) instead of IWI from ViT-T only (ablation) | ViT-B · ImageNet-1K | top-1 / epochs: 92 -> 18 epochs; 31.71 -> 31.83 | Improved |
| use the model chain instead of IWI from ViT-T only (ablation) | ViT-S · ImageNet-1K | top-1 / epochs: 32 -> 24 epochs; 30.11 -> 30.24 | Improved |
| lengthen baseline pretraining from 32 to 128 epochs | LaCLIP baseline · ImageNet-1K (CC3M) | top-1: 28.80 -> 31.80 | Improved |
| replace baseline pretraining with CoM-PT | ViT-L/16 CC3M · COCO | R@1: 36.53 -> 39.18 | Improved |
| replace baseline pretraining with CoM-PT | Swin-L CC3M · ImageNet-1K | top-1: 36.77 -> 37.06 | Improved |
| replace baseline backbone with CoM-PT backbone in SAN | ViT-B/16 · Open-vocab segmentation average | mIoU: 23.53 -> 29.17 | Improved |
| replace baseline backbone with CoM-PT backbone in SAN | ViT-L/16 · Open-vocab segmentation average | mIoU: 24.57 -> 30.71 | Improved |
| replace baseline backbone with CoM-PT backbone in LLaVA-1.5-7B | ViT-B/16 · ScienceQA | accuracy: 62.57 -> 63.41 | No clear effect |
Closest papers by meaning
- Accelerating Pre-training of Multimodal LLMs via Chain-of-Sight (NeurIPS 2024 · similarity 0.89)
- Conformal Prediction for Zero-Shot Models (CVPR 2025 · similarity 0.88)
- Accelerating Augmentation Invariance Pretraining (NeurIPS 2024 · similarity 0.87)
- Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models (ICML 2024 · similarity 0.87)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.87)
- Elastic ViTs from Pretrained Models without Retraining (NeurIPS 2025 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
