Research map / Vision-language and multimodal / CLIP and zero-shot
Bootstrapping SparseFormers from Vision Foundation Models
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Bootstraps SparseFormer vision transformers from pretrained ViT/CLIP weights by inheriting cortex blocks and aligning final token embeddings with cosine loss on ImageNet-1K images only.
Problem
Training SparseFormers from scratch is expensive and hard to scale, and SparseFormer's largest variant reaches only ~84.8% ImageNet top-1, limiting its use as an efficient vision foundation model.
Approach
Inherit the standard transformer (cortex) blocks from large pretrained ViTs, freeze most of them, randomly initialize only the lightweight focusing transformer, and train by aligning SparseFormer's final token embedding with the teacher's [CLS]/projection embedding via cosine loss on unlabeled ImageNet-1K images.
Key results
- SF-LAugReg bootstrapped from ViT-L/16-384AugReg reaches 84.9% IN-1K top-1 with 49 tokens at 1353 img/s.
- SF-LCLIP with 144 tokens reaches 75.9% IN-1K zero-shot accuracy, 0.7 points below CLIP ViT-L/14-336 at 576 tokens, with 33.9 GFLOPs vs 191 GFLOPs.
- Bootstrapping SF-LAugReg from ViT-L/16-224AugReg with 80 tokens reaches 85.2% top-1 at 1008 img/s vs 388 img/s for the teacher, roughly 2.4x throughput.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add classification loss to representation-alignment bootstrapping (ablation) | SF-BAugReg, 49 tokens · ImageNet-1K val | top-1 accuracy: 82.5 -> 77.4 | Hurt |
| reuse pretrained weights in the cortex transformer (ablation) | SF-BAugReg, 49 tokens · ImageNet-1K val | top-1 accuracy: 69.5 -> 82.5 | Helped |
| use representation alignment instead of KL distillation at τ=3 (ablation) | SF-BAugReg, 49 tokens · ImageNet-1K val | top-1 accuracy: 81.0 -> 82.5 | Helped |
| use representation alignment instead of KL distillation at τ=5 (ablation) | SF-BAugReg, 49 tokens · ImageNet-1K val | top-1 accuracy: 81.5 -> 82.5 | Helped |
| bootstrap from pretrained weights rather than train from scratch (ablation) | SparseFormer, 49 tokens · ImageNet-1K val | top-1 accuracy: 80.0 -> 82.5 | Helped |
| increase token count from 49 to 80 (ablation) | SF-BAugReg · ImageNet-1K val | top-1 accuracy: 82.5 -> 83.4 | Helped |
| increase token count from 49 to 80 (ablation) | SF-LAugReg · ImageNet-1K val | top-1 accuracy: 84.5 -> 85.2 | Helped |
| increase token count from 49 to 196 (ablation) | SF-LAugReg · ImageNet-1K val | top-1 accuracy: 84.9 -> 86.7 | Helped |
| increase token count from 64 to 128 (ablation) | SF-LCLIP · ImageNet-1K zero-shot | accuracy: 73.6 -> 74.9 | Helped |
| increase vision encoder token count from 64 to 128 (ablation) | LLaVA with SF-LCLIP · ScienceQA test | multi-modal only accuracy: 84.5 -> 85.8 | Helped |
| replace CLIP ViT-L/14-336 with a bootstrapped SF-LCLIP using 144 tokens | CLIP visual encoder · ImageNet-1K zero-shot | accuracy: 76.6 -> 75.9 | Hurt |
| replace CLIP ViT-L/14-336 with a bootstrapped SF-LCLIP using 144 tokens | CLIP visual encoder · MS COCO val | I→T@1: 57.9 -> 57.0 | Hurt |
Surprising findings
- Adding a classification loss to the alignment objective impaired performance (82.5% to 77.4%) for SF-BAugReg.
- Bootstrapping without reusing pretrained weights yielded inferior results (69.5%) even with 20 epochs of training.
Limitations
- Requires transformer-based vision foundation models and access to their weights, so it cannot apply to non-transformer or proprietary models.
- Bootstrapped models lag behind teachers at 49 tokens; more tokens are needed to close the gap.
- Standard bootstrapping on ImageNet gave weak results on the ScienceQA domain, requiring extra domain-specific bootstrapping.
Experimental setup
Datasets: ImageNet-1K ScienceQA training images
Benchmarks: ImageNet-1K ADE20k MS COCO Flickr30k ScienceQA
Metrics: top-1 accuracy mIoU zero-shot accuracy image-to-text recall@1 text-to-image recall@1 throughput (img/s) FLOPs
Baselines: ViT-B/16-224AugReg ViT-L/16-384AugReg ToMe DiffRate TinyCLIP Swin-L + UPerNet ViT-Adapter CLIP ViT-L/14-336
Builds on: SparseFormer ViT AugReg CLIP LLaVA
Closest papers by meaning
- SparseFormer: Sparse Visual Recognition via Limited Latent Tokens (ICLR 2024 · similarity 0.89)
- SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference (ICML 2025 · similarity 0.88)
- Beyond Interpretability: When, Why, and How Sparse Autoencoders Enable Label-Free Visual Steering (NeurIPS 2026 · similarity 0.86)
- VITRIX-UniViTAR: Unified Vision Transformer with Native Resolution (NeurIPS 2025 · similarity 0.86)
- Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models (NeurIPS 2025 · similarity 0.86)
- ApET: Approximation-Error Guided Token Compression for Efficient VLMs (CVPR 2026 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.