Research map / Architectures and efficiency / Attention and transformers
Can Vision Transformers Learn without Natural Images?
Kodai Nakashima, Hirokatsu Kataoka, Asato Matsumoto, Kenji Iwata, Nakamasa Inoue, Yutaka Satoh
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train on FractalDB-10k instead of FractalDB-1k (ablation) | DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: 96.8 -> 97.6; 81.7 -> 83.5; 86.0 -> 87.7; 98.3 -> 98.8 | Improved |
| Render fractals with patches rather than points (ablation) | FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: +2.6, +4.7, +20.6, +3.2 | Improved |
| Use grayscale rather than color fractal images (ablation) | FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: 97.1 vs. 96.8; 82.6 vs. 81.6; 87.1 vs. 86.0; 98.3 vs. 98.3 | Improved |
| Increase pre-training from 100 to 300 epochs (ablation) | FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: 96.1 -> 96.8; 81.1 -> 81.6; 82.0 -> 86.0; 96.5 -> 98.3 | Mixed |
| Use patch size 16 rather than 32 (ablation) | FractalDB-1k-pretrained DeiT · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: 96.8 vs. 95.5; 81.6 vs. 78.4; 86.0 vs. 76.0; 98.3 vs. 95.7 | Improved |
| Pre-train with FractalDB-1k rather than from scratch | DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: +18.5, +23.9, +74.4, +21.2 | Improved |
| Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2 | DeiT-Ti/16 · CIFAR-10, Stanford Cars, Pascal VOC 2012 | accuracy: 97.6 vs. 97.4; 87.7 vs. 84.9; 86.9 vs. 86.2 | Improved |
| Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2 | DeiT-Ti/16 · CIFAR-100, Flowers-102, Places-30 | accuracy: 83.5 vs. 84.1; 98.8 vs. 98.9; 78.5 vs. 80.0 | Hurt |
| Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2 | DeiT-Ti/16 · Average across representative datasets | accuracy: 88.5 -> 88.8 | Improved |
| Use FractalDB rather than BezierCurveDB pre-training | DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: +0.1, +1.3, +3.2, -0.2 | Mixed |
| Use FractalDB-1k pre-training rather than ResNet-50 with FractalDB-1k | DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102 | accuracy: 96.8 vs. 96.1; 81.6 vs. 80.0; 86.0 vs. 82.5; 98.3 vs. 98.2 | Improved |
Closest papers by meaning
- Replacing Labeled Real-Image Datasets With Auto-Generated Contours (CVPR 2022 · similarity 0.90)
- Can You Learn to See Without Images? Procedural Warm-Up for Vision Transformers (CVPR 2026 · similarity 0.90)
- DeiT III: Revenge of the ViT (ECCV 2022 · similarity 0.89)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.86)
- Non-Natural Image Understanding with Advancing Frequency-based Vision Encoders (CVPR 2025 · similarity 0.86)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
