Research map / Architectures and efficiency / Attention and transformers
An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that vanilla Transformers treating each individual pixel as a token (with learned position embeddings) remove locality bias and match or beat patch-based ViT/DiT baselines on CIFAR-100, ImageNet, Flowers, NYU-v2 and ImageNet diffusion generation.
Problem
Whether the inductive bias of locality (neighboring pixels being more related, enforced via patchification and position embeddings) is necessary for computer vision Transformers.
Approach
Treat each pixel as a token for a vanilla Transformer with learned, content-agnostic position embeddings, removing all 2D locality priors; the paper finds this matches or outperforms 2x2 or 16x16 patch-based ViT and DiT at comparable model sizes, with no new method proposed.
Key results
- On CIFAR-100, ViT-T/1 (pixels) reaches 85.1% Acc@1 vs 83.6% for ViT-T/2, and ViT-S/1 reaches 86.4% vs 83.7% for ViT-S/2.
- On ImageNet at 64x64 input (ViT-L), the pixel variant reaches 76.9% Acc@1 vs 75.6% for ViT-L/2.
- DiT-L/1 (pixel variant) achieves FID 4.05 vs 4.16 for DiT-L/2 at 400 epochs, and FID 2.68 vs 2.89 at 1400 epochs.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use 1×1 pixel tokens instead of 2×2 patches | ViT-T · CIFAR-100 | Acc@1: 83.6 -> 85.1 | Helped |
| Use 1×1 pixel tokens instead of 2×2 patches | ViT-S · ImageNet | Acc@1: 72.9 -> 74.1 | Helped |
| Use 1×1 pixel tokens instead of 2×2 patches | ViT-S · NYU-v2 | RMSE: 0.80 -> 0.72 | Helped |
| Use pixel tokens with MAE pre-training | ViT-T · CIFAR-100 fine-tuning | Acc@1: 85.7 -> 86.0 | Helped |
| Use 1×1 latent tokens instead of 2×2 patches | DiT-L · ImageNet class-conditional generation | FID: 4.16 -> 4.05 | Helped |
| Remove position embeddings (ablation) | ViT-B · ImageNet supervised classification | Acc@1: 82.7 -> 81.2 | Hurt |
| Replace parametric pixel projection with non-parametric dictionary embeddings (ablation) | ViT-B/1 · ImageNet supervised classification | Acc@1: 76.1 -> 76.6 | Helped |
| Use single-pixel rather than bicubic downsizing (ablation) | ViT-B/2 · ImageNet supervised classification | Acc@1: 75.7 -> 70.5 | Hurt |
| Use single-pixel rather than bicubic downsizing (ablation) | ViT-B/1 · ImageNet supervised classification | Acc@1: 76.1 -> 75.6 | Hurt |
| Replace patchification with FH segmentation tokens (ablation) | ViT-B · ImageNet supervised classification | Acc@1: 82.7 -> 81.3 | Hurt |
| Replace patchification with SLIC tokens (2 iterations) (ablation) | ViT-B · ImageNet supervised classification | Acc@1: 82.7 -> 78.9 | Hurt |
Surprising findings
- Replacing position embeddings with none only drops ViT Acc@1 by 1.5%, while pixel permutation drops accuracy by 25.2%, suggesting patchification is the key locality prior.
- A dictionary-based non-parametric pixel embedding outperforms the parametric pixel projection on ImageNet (76.6% vs 76.1%).
Limitations
- Quadratic self-attention cost makes pixel tokens computationally expensive and impractical at scale.
- Experiments use low resolutions (e.g., 28x28 or 64x64 on ImageNet) due to compute budget, and the study is not comprehensive across tasks.
- Training of pixel-based ViT is less stable, requiring reduced learning rates.
Experimental setup
Datasets: CIFAR-100 ImageNet Oxford-102-Flowers NYU-v2
Benchmarks: CIFAR-100 ImageNet Oxford-102-Flowers NYU-v2 FID/sFID/IS on ImageNet generation
Metrics: Acc@1 Acc@5 RMSE RAE FID sFID IS precision recall
Baselines: ViT-T/2 ViT-S/2 ViT-B/2 ViT-L/2 DiT-L/2 DiT-XL/2 iGPT
Builds on: Vision Transformer MAE DiT VQGAN
Closest papers by meaning
- Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (NeurIPS 2021 · similarity 0.90)
- Revisiting [CLS] and Patch Token Interaction in Vision Transformers (ICLR 2026 · similarity 0.89)
- Vision Transformers Need Registers (ICLR 2024 · similarity 0.89)
- Fine-Tuning Image Transformers Using Learnable Memory (CVPR 2022 · similarity 0.89)
- AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (CVPR 2022 · similarity 0.89)
- A Theoretical Analysis of Self-Supervised Learning for Vision Transformers (ICLR 2025 · similarity 0.88)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.