Research map / Architectures and efficiency / Pruning and efficient tuning
Visual Atoms: Pre-Training Vision Transformers With Sinusoidal Waves
Sora Takashima, Ryo Hayamizu, Nakamasa Inoue, Hirokatsu Kataoka, Rio Yokota
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Pre-train with VisualAtom-1k instead of RCDB-1k | ViT-T · CIFAR100 | accuracy: 82.2 -> 84.9 | Improved |
| Pre-train with VisualAtom-21k instead of RCDB-21k | ViT-B · CIFAR100 | accuracy: 82.9 -> 86.7 | Improved |
| Scale pre-training from VisualAtom-1k to VisualAtom-21k | ViT-B · Average of seven evaluation datasets | accuracy: 89.1 -> 89.7 | Improved |
| Pre-train with VisualAtom-1k instead of ImageNet-21k | ViT-T · ImageNet-1k fine-tuning, 224² | top-1 accuracy: 74.1 -> 74.2 | Improved |
| Use VisualAtom-21k rather than JFT-300M for pre-training | ViT-B · ImageNet-1k fine-tuning, 384² | top-1 accuracy: 84.2 -> 83.7 | Hurt |
| Increase the maximum frequency beyond 20 (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.9 -> 84.3; 84.9 -> 84.1 | Hurt |
| Narrow the frequency range by raising its minimum (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.9 -> 84.8; 84.9 -> 84.4; 84.9 -> 83.6 | Hurt |
| Increase variation in orbit count from fixed/high counts to a broad range (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.3 -> 84.9 | Improved |
| Use low quantization values (3–200) rather than the baseline range (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.9 -> 84.6 | Hurt |
| Vary amplitude rather than fixing it at 0.5 (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.9 -> 84.9 | No clear effect |
| Increase fixed amplitude from 0.5 to 1.0 (ablation) | VisualAtom-1k · CIFAR100 | accuracy: 84.9 -> 84.4 | Hurt |
Closest papers by meaning
- Replacing Labeled Real-Image Datasets With Auto-Generated Contours (CVPR 2022 · similarity 0.94)
- Pre-training Vision Models with Mandelbulb Variations (CVPR 2024 · similarity 0.88)
- Linearly Decomposing and Recomposing Vision Transformers for Diverse-Scale Models (NeurIPS 2024 · similarity 0.84)
- Non-Natural Image Understanding with Advancing Frequency-based Vision Encoders (CVPR 2025 · similarity 0.84)
- Oscillation-Reduced MXFP4 Training for Vision Transformers (ICML 2025 · similarity 0.84)
- Visual Fourier Prompt Tuning (NeurIPS 2024 · similarity 0.84)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
