Research map / Detection and segmentation / CNN features
Semantic-Aware Autoregressive Image Modeling for Visual Representation Learning
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes SemAIM, a self-supervised autoregressive image model that orders ViT patches from semantic center outward (using [CLS]-patch similarity) and predicts DINO/CLIP patch features rather than raw pixels.
Problem
Autoregressive image modeling for self-supervised visual pre-training lags behind masked image modeling and NLP because images lack a natural sequential order; raster and random patch orders are not aligned with how humans attend to images.
Approach
Generate a semantic-aware permutation by ranking patches by their similarity to the [CLS] token (smoothed, with a center-to-outward distance ordering plus small random jitter), run autoregression with a parallel encoder-decoder using causal masks, and predict pretrained DINO or CLIP patch features instead of raw pixels.
Key results
- With ViT-B/16 and 800 epochs of pre-training, SemAIM achieves 84.1% ImageNet fine-tuning top-1, 51.3 AP^b and 45.4 AP^m on COCO, outperforming MAE by 0.5, 1.0 and 0.5 points.
- Using CLIP features as prediction targets, SemAIM reaches 85.3% (ViT-B) and 86.5% (ViT-L) ImageNet fine-tuning accuracy.
- The semantic-aware order (83.5 fine-tune) beats raster (82.9) and stochastic (83.0) orders in the ablation; the raw similarity order is worst on dense tasks (38.9 AP^b).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use semantic-aware order instead of raster order (ablation) | SemAIM · ImageNet classification | top-1 accuracy: 82.9 -> 83.5 | Helped |
| Use semantic-aware order instead of stochastic order (ablation) | SemAIM · COCO object detection | APb: 42.1 -> 43.3 | Helped |
| Use the similarity map directly as the autoregression order (ablation) | SemAIM · ImageNet classification | top-1 accuracy: 83.5 -> 82.4 | Hurt |
| Use DINO features instead of RGB pixels as prediction targets (ablation) | SemAIM · ImageNet classification | top-1 accuracy: 83.5 -> 84.4 | Helped |
| Use CLIP features instead of RGB pixels as prediction targets (ablation) | SemAIM · ImageNet classification | top-1 accuracy: 83.5 -> 84.6 | Helped |
| Increase decoder depth from 1 to 12 (ablation) | SemAIM · ImageNet classification | top-1 accuracy: 82.1 -> 83.5 | Helped |
| Increase pre-training from 400 to 800 epochs | SemAIM · ImageNet-1K fine-tune | top-1 accuracy: 83.8 -> 84.1 | Helped |
| Use semantic-aware autoregression instead of SAIM | ViT-B/16 pre-training · COCO object detection | APb: +1.9% | Helped |
| Use semantic-aware autoregression instead of SAIM | ViT-B/16 pre-training · COCO instance segmentation | APm: +1.4% | Helped |
| Use semantic-aware autoregression instead of SAIM | ViT-B/16 pre-training · ADE20k semantic segmentation | mIoU: +0.2% | Helped |
Surprising findings
- Directly using the similarity map as the autoregression order performs worst, below raster and stochastic orders on COCO detection and segmentation.
- Autoregressive image modeling needs a deeper decoder (12 blocks) than masked image modeling; decoder depth 1 drops ImageNet fine-tune to 82.1.
Limitations
- Permutation generation uses only one center patch, which may not handle images with multiple objects well.
- Uses features from pretrained DINO or CLIP ViT-B as targets, so it depends on an external teacher model.
Experimental setup
Datasets: ImageNet-1K COCO ADE20k
Benchmarks: ImageNet-1K COCO ADE20k
Metrics: top-1 accuracy AP^b AP^m mIoU k-NN accuracy linear probing accuracy
Baselines: MAE SAIM RandSAC iGPT ViT-iGPT DINO MoCo v3 BEiT iBOT BootMAE HPM SimMIM
Builds on: SAIM MAE iGPT ViTDet DINO
Closest papers by meaning
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.88)
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.87)
- Seg-VAR:Image Segmentation with Visual Autoregressive Modeling (NeurIPS 2025 · similarity 0.87)
- Scalable Pre-training of Large Autoregressive Image Models (ICML 2024 · similarity 0.87)
- Vision Foundation Models as Effective Visual Tokenizers for Autoregressive Generation (NeurIPS 2025 · similarity 0.85)
- FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction (NeurIPS 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.