Research map / Architectures and efficiency / Attention and transformers
Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers
Zachary Shinnick, Liangze Jiang, Hemanth Saratchandran, Damien Teney, Anton van den Hengel
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| add procedural k-Dyck warm-up before image training | ViT-B/16 · ImageNet-1k (from scratch) | accuracy (%): 77.49 -> 79.21 (+1.72) | Improved |
| add procedural k-Dyck warm-up instead of default init | ViT-T/16 · CIFAR-100 (from scratch) | accuracy (%): 68.52 -> 71.98 | Improved |
| use FractalDB visual warm-up instead of no warm-up | ViT-T/16 · CIFAR-100 (from scratch) | accuracy (%): 68.52 -> 64.61 | Hurt |
| add procedural warm-up before ImageNet-1k pretraining, then fine-tune | ViT-B/16 · CIFAR-100 (fine-tuned) | accuracy (%): 87.54 -> 89.20 | Improved |
| replace 1% of pretraining budget with procedural data | ViT-B/16 · ImageNet-1k | accuracy (%): 28% fewer image examples for same accuracy | Improved |
| use k-Dyck Shuffle grammar rather than k-Dyck (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 71.98 -> 70.11 | Hurt |
| use WW grammar rather than k-Dyck (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 68.52 -> 66.44 | Hurt |
| keep sequence token order intact (vs shuffled tokens) (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 67.22 -> 71.98 | Improved |
| keep weight structure (vs shuffling all weights) (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 69.51 -> 71.98 | Improved |
| keep attention weights intact (vs shuffled) (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 70.57 -> 71.98 | Improved |
| keep MLP weights intact (vs shuffled) (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 70.71 -> 71.98 | Improved |
| transfer all layers rather than only first 4 (ablation) | procedural warm-up · CIFAR-100 (from scratch) | accuracy (%): 68.91 -> 71.98 | Improved |
Closest papers by meaning
- Can Vision Transformers Learn without Natural Images? (AAAI 2022 · similarity 0.90)
- On the Surprising Effectiveness of Attention Transfer for Vision Transformers (NeurIPS 2024 · similarity 0.89)
- DeiT III: Revenge of the ViT (ECCV 2022 · similarity 0.89)
- Vision Transformers with Self-Distilled Registers (NeurIPS 2025 · similarity 0.88)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.87)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.87)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
