Research map / Architectures and efficiency / Attention and transformers
Scaling Laws in Patchification: An Image Is Worth 50,176 Tokens And More
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that shrinking the patchification patch size toward pixel tokenization consistently improves ViT and Mamba (Adventurer) performance across classification, segmentation and detection, reaching 84.6% ImageNet accuracy with a 50,176-token base model.
Problem
Patchification compresses images into few tokens, causing information loss in plain visual architectures; the paper asks whether reducing patch size (toward pixel tokens) is a scalable dimension that improves visual understanding.
Approach
Treats patch size as a new scaling dimension: systematically reducing patch size down to 1x1 (pixel tokenization) yields smooth, consistent gains across tasks, resolutions and architectures, and decoder heads become less necessary for dense prediction at low patch sizes.
Key results
- Adventurer-Base at 224x224 improves ImageNet top-1 from 82.6% (16x16 patch, 196 tokens) to 84.6% with pixel tokenization (50,176 tokens).
- On ADE20k with decoder-free linear head, Adventurer-B mIoU rises from 44.0 (16x16) to 46.8 (2x2).
- On COCO with Adventurer-Base, box AP improves from 44.1 (64x64) to 50.3 (4x4) and mask AP from 38.3 to 43.4.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Reduce patch size from 16×16 to 1×1 | Adventurer-Base · ImageNet-1k, 224×224 | accuracy: 82.6 -> 84.6 | Helped |
| Remove UperNet and reduce patch size from 16×16 to 2×2 | Adventurer-B · ADE20k semantic segmentation | mIoU: 45.7 -> 46.8 | Helped |
| Remove UperNet and reduce patch size from 16×16 to 2×2 | Adventurer-T · ADE20k semantic segmentation | mIoU: 41.3 -> 42.5 | Helped |
| Reduce patch size from 64×64 to 4×4 | Adventurer-B · COCO object detection | APb: 44.1 -> 50.3 | Helped |
| Reduce patch size from 64×64 to 4×4 | Adventurer-B · COCO instance segmentation | APm: 38.3 -> 43.4 | Helped |
| Extend sequence length by interpolating existing tokens instead of reducing patch size (ablation) | DeiT-Base · ImageNet-1k, 128×128 | accuracy: 78.1 -> 78.2 | No clear effect |
| Reduce patch size to 4×4 rather than interpolate existing tokens (ablation) | DeiT-Base · ImageNet-1k, 128×128 | accuracy: 78.1 -> 82.3 | Helped |
| Extend sequence length by interpolating existing tokens instead of reducing patch size (ablation) | Adventurer-Base · ImageNet-1k, 224×224 | accuracy: 82.6 -> 82.8 | No clear effect |
| Reduce patch size to 2×2 rather than interpolate existing tokens (ablation) | Adventurer-Base · ImageNet-1k, 224×224 | accuracy: 82.6 -> 84.5 | Helped |
| Increase input resolution beyond the original image resolution while fixing sequence length (ablation) | ImageNet classification models · ImageNet-1k | accuracy | Hurt |
| Scale up model parameter count (ablation) | Adventurer · ImageNet classification | accuracy | Mixed |
Surprising findings
- Pixel-level tokenization of a 224x224 ImageNet image (50,176 tokens) still improves accuracy, while merely extending the token sequence via interpolation gives almost no gain.
- With small patches, a simple linear segmentation head performs competitively with a UperNet decoder, suggesting decoders become less critical.
Limitations
- ViT quadratic cost restricts it to sequences up to about 4,096 tokens, so the pixel-level ViT scaling is limited by compute and memory.
- Parameter scaling beyond about 760M parameters yields no further gains, and input-size scaling is capped by original image resolution.
- Results are reported mainly for a limited set of architectures (ViT and Adventurer) and tasks.
Experimental setup
Datasets: ImageNet-1k ADE20k COCO
Benchmarks: ImageNet-1k ADE20k COCO
Metrics: top-1 accuracy mIoU box AP mask AP test loss
Baselines: DeiT-Base at 16x16 patch Adventurer-Base at 16x16 patch UperNet decoder parameter scaling input size scaling token interpolation (sequence extension)
Builds on: Vision Transformer (ViT) DeiT Adventurer Mamba Cascade Mask R-CNN UperNet
Closest papers by meaning
- Learning Hierarchical Patch Splitting Policies for Faster Vision Transformers (NeurIPS 2026 · similarity 0.91)
- Patch-as-Decodable-Token: Towards Unified Multi-Modal Vision Tasks in MLLMs (ICLR 2026 · similarity 0.87)
- MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution (NeurIPS 2024 · similarity 0.87)
- Learnings from Scaling Visual Tokenizers for Reconstruction and Generation (ICML 2025 · similarity 0.87)
- REOrdering Patches Improves Vision Models (NeurIPS 2025 · similarity 0.86)
- An Image is Worth More Than 16x16 Patches: Exploring Transformers on Individual Pixels (ICLR 2025 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.