Research map / Label-efficient and robust learning / Self-supervised and contrastive
Generative Pretraining From Pixels
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's title only (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| use best intermediate layer instead of final layer for linear probing (ablation) | iGPT-L · CIFAR-10 | linear probe accuracy: 2.4% higher than final layer | Helped |
| scale model size (iGPT-S to iGPT-M to iGPT-L) (ablation) | iGPT generative pretraining · ImageNet linear probe | best-layer top-1 accuracy: 41.9 (S) -> 54.5 (M) -> 60.3 (L) | Helped |
| BERT objective instead of auto-regressive objective (ablation) | iGPT-L · CIFAR-10 linear probe and fine-tune | linear probe and fine-tune accuracy: best-layer linear probe >1% lower; fine-tune 98.6 vs 99.0 | Hurt |
| BERT objective instead of auto-regressive objective (ablation) | iGPT-L · ImageNet | linear probe and fine-tune accuracy: linear probe 6% lower; fine-tune 66.5 slightly above AR | Mixed |
| ensemble predictions over 5 BERT masks at evaluation (ablation) | iGPT-L BERT · ImageNet | linear probe and fine-tune accuracy: almost 1% | Helped |
| ensemble predictions over 5 BERT masks at evaluation (ablation) | iGPT-L BERT · CIFAR-10 | linear probe and fine-tune accuracy: largely unchanged | No clear effect |
| increase model resolution from 32^2 to 48^2 (concatenating layers) (ablation) | iGPT-L · ImageNet linear probe | best-layer top-1 accuracy: 60.3 -> 65.2 | Helped |
| VQ-VAE encoding at IR 192^2x3 instead of clustering pixels (ablation) | iGPT-L · ImageNet linear probe | best-layer top-1 accuracy: 65.2 -> 65.3 best-layer; 69.0 with 11-layer concat vs 67.3 without | No clear effect |
| concatenate features from 11 layers centered at best layer (ablation) | iGPT-L · ImageNet linear probe | top-1 accuracy: 65.2 -> 67.3 (MR 48) and 65.3 -> 69.0 (VQ-VAE IR 192) | Helped |
| add joint generative + classification loss during fine-tuning | iGPT-L · fine-tuning | accuracy | Helped |
| dropout regularization during fine-tuning | iGPT-L · fine-tuning | accuracy | No clear effect |
| larger model capacity improves linear probe accuracy at equal validation loss | iGPT · CIFAR-10 linear probe | linear probe accuracy | Helped |
Closest papers by meaning
- Rejuvenating image-GPT as Strong Visual Representation Learners (ICML 2024 · similarity 0.89)
- Advancing End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-Training (ICLR 2026 · similarity 0.89)
- Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction? (ICML 2026 · similarity 0.88)
- Can Generative Models Improve Self-Supervised Representation Learning? (AAAI 2025 · similarity 0.87)
- Boosting Generative Image Modeling via Joint Image-Feature Synthesis (NeurIPS 2025 · similarity 0.86)
- PixelDiT2: Representation-Grounded Pixel Diffusion Transformers (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.