AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Can Vision Transformers Learn without Natural Images?

Kodai Nakashima, Hirokatsu Kataoka, Asato Matsumoto, Kenji Iwata, Nakamasa Inoue, Yutaka Satoh

AAAI 2022 · Architectures and efficiency › Attention and transformers · approach: Hybrid and interpretable ViT designs · 35 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
Pre-train on FractalDB-10k instead of FractalDB-1k (ablation)DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: 96.8 -> 97.6; 81.7 -> 83.5; 86.0 -> 87.7; 98.3 -> 98.8Improved
Render fractals with patches rather than points (ablation)FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: +2.6, +4.7, +20.6, +3.2Improved
Use grayscale rather than color fractal images (ablation)FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: 97.1 vs. 96.8; 82.6 vs. 81.6; 87.1 vs. 86.0; 98.3 vs. 98.3Improved
Increase pre-training from 100 to 300 epochs (ablation)FractalDB-pretrained DeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: 96.1 -> 96.8; 81.1 -> 81.6; 82.0 -> 86.0; 96.5 -> 98.3Mixed
Use patch size 16 rather than 32 (ablation)FractalDB-1k-pretrained DeiT · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: 96.8 vs. 95.5; 81.6 vs. 78.4; 86.0 vs. 76.0; 98.3 vs. 95.7Improved
Pre-train with FractalDB-1k rather than from scratchDeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: +18.5, +23.9, +74.4, +21.2Improved
Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2DeiT-Ti/16 · CIFAR-10, Stanford Cars, Pascal VOC 2012accuracy: 97.6 vs. 97.4; 87.7 vs. 84.9; 86.9 vs. 86.2Improved
Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2DeiT-Ti/16 · CIFAR-100, Flowers-102, Places-30accuracy: 83.5 vs. 84.1; 98.8 vs. 98.9; 78.5 vs. 80.0Hurt
Pre-train on FractalDB-10k rather than ImageNet-1k with SimCLRv2DeiT-Ti/16 · Average across representative datasetsaccuracy: 88.5 -> 88.8Improved
Use FractalDB rather than BezierCurveDB pre-trainingDeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: +0.1, +1.3, +3.2, -0.2Mixed
Use FractalDB-1k pre-training rather than ResNet-50 with FractalDB-1kDeiT-Ti/16 · CIFAR-10, CIFAR-100, Stanford Cars, Flowers-102accuracy: 96.8 vs. 96.1; 81.6 vs. 80.0; 86.0 vs. 82.5; 98.3 vs. 98.2Improved

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy