Research map / Architectures and efficiency / CNN design
Bootstrapping ViTs: Towards Liberating Vision Transformers From Pre-Training
Check your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add joint agent-CNN training without weight sharing | ViT-S · CIFAR-10 | top-1 accuracy: 87.32 -> 95.14 | Helped |
| Add joint agent-CNN training without weight sharing | ViT-S · CIFAR-100 | top-1 accuracy: 61.25 -> 76.19 | Helped |
| Use joint training without weight sharing rather than shared-weight bootstrapping | ViT-B · ImageNet-1k, 10% images | accuracy: 45.75 -> 49.73 | Mixed |
| Add adaptive intermediate feature supervision (ablation) | ViT-S joint training · CIFAR-100 | accuracy: 61.25 -> 74.94 | Helped |
| Add mutual knowledge distillation (ablation) | ViT-S with feature supervision · CIFAR-100 | accuracy: 74.94 -> 76.19 | Helped |
| Decay feature-supervision strength during training rather than keep it constant (ablation) | ViT-S with both loss terms · CIFAR-100 | accuracy: 75.15 -> 76.19 | Helped |
| Use sequence interpolation instead of 2D average pooling for feature adaptation (ablation) | ViT-S with feature supervision only · CIFAR-100 | accuracy: 71.06 -> 74.94 | Helped |
| Use a res-like agent rather than a base agent (ablation) | ViT-S joint training · CIFAR-100 | accuracy: 73.18 -> 76.19 | Helped |
Closest papers by meaning
- Structured Initialization for Vision Transformers (NeurIPS 2025 · similarity 0.89)
- Bootstrapping SparseFormers from Vision Foundation Models (CVPR 2024 · similarity 0.88)
- DeiT-LT: Distillation Strikes Back for Vision Transformer Training on Long-Tailed Datasets (CVPR 2024 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.88)
- GradQ-ViT: Robust and Efficient Gradient Quantization for Vision Transformers (AAAI 2025 · similarity 0.88)
- Slicing Vision Transformer for Flexible Inference (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.