Research map / Label-efficient and robust learning / Self-supervised and contrastive
Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Gives the first theoretical comparison of autoregressive vs. masked generative SSL, showing masked SSL gives better linear classification via sample connectivity and autoregressive SSL better generation due to length misalignment, and proposes diversity-enhanced AR and variable-length masked objectives that improve each.
Problem
Theoretically understanding why autoregressive and masked generative self-supervised pretraining differ in downstream classification and content generation performance.
Approach
Reformulates the generative SSL spectral loss as asymmetric matrix factorization of the co-occurrence matrix, linking downstream classification error to its singular values (connectivity); shows masked targets yield more connections, and that masked SSL's fixed unmasked length misaligns with variable test lengths in generation. Proposes diversity-enhanced AR (predict next t tokens) and variable-ma
Key results
- Diversity-enhanced autoregressive objective raises GLUE average from 74.1 to 76.0 and ViT-S linear probing from 33.1% to 36.2%.
- Variable-length masked objective lowers Pile perplexity from 50.2 (fixed 0.3 mask ratio) to 45.1 and C4 perplexity from 63.8 to 59.4.
- Variable-length masked objective reduces ImageNet reconstruction L2 loss from 0.115 to 0.072 relative to fixed mask ratio 0.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Use masked rather than autoregressive pretraining | Generative SSL models · GLUE | GLUE score: 72.8 -> 80.5 | Helped |
| Use autoregressive rather than masked pretraining | Generative SSL models · WikiText-2 | perplexity: 34.1 -> 22.8 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · GLUE average | score: 74.1 -> 76.0 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · GLUE SST-2 | accuracy: 90.3 -> 89.9 | Hurt |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-100, ViT-S | linear probing accuracy: 33.1 -> 36.2 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-100, ViT-S | fine-tuning accuracy: 81.1 -> 83.2 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-1K, ViT-B | linear probing accuracy: 56.2 -> 59.4 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-1K, ViT-B | fine-tuning accuracy: 82.5 -> 82.9 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-1K few-shot, ViT-B | linear probing accuracy: 41.2 -> 45.3 | Helped |
| Use diversity-enhanced autoregressive objective | Autoregressive model · ImageNet-1K few-shot, ViT-B | fine-tuning accuracy: 70.2 -> 71.1 | Helped |
| Use variable-length masking with ratios sampled from [0.15,0.3] (ablation) | Masked model · Pile | perplexity: 59.6 -> 45.1; 50.2 -> 45.1 | Helped |
| Use variable-length masking with ratios sampled from [0.15,0.3] (ablation) | Masked model · C4 | perplexity: 71.2 -> 59.4; 63.8 -> 59.4 | Helped |
Surprising findings
- Masked SSL underperforms autoregressive SSL on content generation specifically for short conditional texts, attributed to fixed unmasked length misalignment.
Limitations
- Theory relies on simplifying assumptions: spectral loss surrogate, linear attention encoder, and negligible labeling error within samples.
- Connectivity is estimated via feature similarity surrogate rather than the true co-occurrence matrix on real data.
- Vision experiments reach low absolute accuracy (e.g., ViT-S LP 33-36%) and fine-tuning gains for ViT-B are small (0.4%).
Experimental setup
Datasets: The Pile C4 GLUE ImageNet-1K ImageNet-100 WikiText-2 FGVC Aircraft Caltech-101 Stanford Cars CIFAR10 CIFAR100 DTD
Benchmarks: GLUE WikiText-2 The Pile (held-out tail) C4 ImageNet-1K ImageNet-100 9 transfer datasets
Metrics: GLUE score perplexity linear probing accuracy fine-tuning accuracy L2 loss perceptual (LPIPS) loss
Baselines: autoregressive SSL masked SSL MAE BERT
Builds on: GPT BERT XLNet MAE Cramming
Closest papers by meaning
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.88)
- Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More (ICML 2025 · similarity 0.86)
- Clustering Properties of Self-Supervised Learning (ICML 2025 · similarity 0.86)
- DiffScore: Text Evaluation Beyond Autoregressive Likelihood (NeurIPS 2026 · similarity 0.86)
- Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks (ICML 2024 · similarity 0.86)
- Hierarchical Masked Autoregressive Models with Low-Resolution Token Pivots (ICML 2025 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.