Research map / Vision-language and multimodal / Human and embodied modeling
Rejuvenating image-GPT as Strong Visual Representation Learners
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Rejuvenates iGPT by autoregressively predicting CLIP-derived semantic tokens of clusters of image patches plus a discriminative decoder that reconstructs visible-cluster tokens, reaching 90.0% ImageNet-1K top-1 with ViT-H pretrained on public data.
Problem
Autoregressive pixel prediction (iGPT) lags behind BERT-style masked image modeling for visual representation learning, and its quadratic cost limits it to low resolution.
Approach
Shift the autoregressive target from raw pixels to semantic tokens from a discriminatively trained teacher (CLIP), operate on spatial clusters of patches inside a ViT, and add a separate discriminative decoder that distills teacher tokens of already-visible clusters, with the two decoders disentangling the supervisions.
Key results
- D-iGPT ViT-B pretrained on ImageNet-1K for 300 epochs reaches 86.2% ImageNet-1K top-1 and 53.8 mIoU on ADE20K, surpassing MILAN/DeepMIM by 0.6%.
- D-iGPT ViT-H pretrained on ImageNet-21K reaches 90.0% ImageNet-1K top-1 using public data, with ViT-L at 89.5%.
- Using CLIP tokens cuts training cost by 77% (666h to 159h) versus MAE with CLIP targets while improving accuracy from 85.4% to 86.2%.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| replace pixel targets with CLIP semantic tokens (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.4 to 86.2; +0.8 | Helped |
| use DINO semantic tokens instead of the MAE target (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 84.4 to 84.7; +0.3 | Helped |
| use CLIP semantic tokens instead of pixels for autoregressive prediction (ablation) | iGPT · ImageNet-1K classification | top-1 accuracy: 82.0 to 86.2 | Helped |
| increase tokenizer from CLIP-B to CLIP-L (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.7 to 85.9 | Helped |
| use CLIP-L@336 tokenizer at 224×224 input resolution (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.9 to 84.6 | Hurt |
| scale tokenizer from OpenCLIP-L to OpenCLIP-H (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.9 to 86.2 | Helped |
| increase cluster count from 1 to 4 (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.3 to 86.2 | Helped |
| increase cluster count beyond 4, up to 196 (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 86.2 to 85.4 | Hurt |
| use D-iGPT autoregressive pretraining rather than MAE, EVA, or KD (ablation) | ViT-B representation learning · ImageNet-1K classification | top-1 accuracy: 85.0 to 86.2 | Helped |
| use a 2-layer instead of 1-layer decoder (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 85.6 to 86.2 | Helped |
| use a 2-layer instead of 4-layer decoder (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 86.0 to 86.2 | Helped |
| reduce decoder dimension from 1024 to 512 (ablation) | D-iGPT · ImageNet-1K classification | top-1 accuracy: 86.2 to 85.8 | Hurt |
Surprising findings
- A 2-layer decoder outperforms a 4-layer decoder, unlike MAE's default 8-layer decoder.
- Using CLIP-L at 336 resolution as tokenizer while training at 224 significantly degrades performance, attributed to resolution mismatch.
Limitations
- Relies on a strong discriminatively trained teacher (CLIP/OpenCLIP) to generate semantic tokens; weaker teachers such as DINO yield smaller gains.
- Gains from fine-tuning-based evaluation depend on large-scale pretraining for the best ViT-H result, which is not on ImageNet-1K alone.
Experimental setup
Datasets: ImageNet-1K ImageNet-21K LAION-400M ADE20K ImageNet-V2 ImageNet-Real ImageNet-A ImageNet-R ImageNet-C ImageNet-Sketch ImageNet-Hard
Benchmarks: ImageNet-1K ADE20K ImageNet-V2 ImageNet-Real ImageNet-A ImageNet-R ImageNet-C ImageNet-Sketch ImageNet-Hard
Metrics: top-1 accuracy mIoU linear probing accuracy zero-shot top-1 accuracy training cost (hours)
Baselines: MAE BEiT BEiTv2 EVA MILAN DeepMIM iBOT TinyMIM One-Peace BEiT-3
Builds on: iGPT BEiT MAE CLIP SAIM ViT
Closest papers by meaning
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.87)
- Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (NeurIPS 2024 · similarity 0.87)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- UVU: Improving Multimodal Understanding via Vision-Language Unified Autoregressive Paradigm (CVPR 2026 · similarity 0.87)
- Multimodal Autoregressive Pre-training of Large Vision Encoders (CVPR 2025 · similarity 0.86)
- Decomposing and Interpreting Image Representations via Text in ViTs Beyond CLIP (NeurIPS 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.