Research map / Architectures and efficiency / Attention and transformers
Self-supervision through Random Segments with Autoregressive Coding (RandSAC)
Tianyu Hua, Yonglong Tian, Sucheng Ren, Michalis Raptis, Hang Zhao, Leonid Sigal
Read the paperCheck your idea against 137,217 papers
AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| randomized serialization of patches instead of raster order (ablation) | patch tokenization, autoregressive ViT · CIFAR10 and ImageNet100 | linear probing and fine-tuning accuracy: CIFAR10 LIN +20 points (55.53 to 75.53); IN100 LIN 49.35 to 53.02 | Improved |
| patch tokenization (P=4) instead of pixel tokenization (ablation) | raster-order autoregressive model · CIFAR10 | linear probing and fine-tuning accuracy: LIN 41.70 to 55.53; FT 59.35 to 78.67 | Improved |
| grouping patches into square segments (M=2) with random serialization (ablation) | patch-random autoregression · CIFAR10 | linear probing and fine-tuning accuracy: LIN 75.53 to 81.38; FT 87.52 to 91.38 | Improved |
| grouping patches into blob segments (K=5) instead of single patches (ablation) | patch-random autoregression · CIFAR10 and ImageNet100 | linear and fine-tune accuracy: CIFAR10 LIN 75.53 to 82.52; IN100 LIN 53.02 to 65.00 | Improved |
| making segments spatially incoherent (random shuffle of tokens) (ablation) | blob segments · CIFAR10 | linear probing: LIN ~80.9-82.5 coherent vs ~76.7-78.6 shuffled | Hurt |
| two-level hierarchical segment serialization (16->4) instead of flat 16 square segments (ablation) | square RandSAC · CIFAR10 and ImageNet100 | linear and fine-tune accuracy: CIFAR10 LIN 81.38 to 82.46; IN100 LIN 64.90 to 65.81 | Improved |
| trainable skip connections from encoder layers to decoder layers (ablation) | masked transformer decoder for RandSAC · CIFAR10 and ImageNet100 | linear probing: CIFAR10 LIN 88.0 to 89.5; IN100 LIN 65.8 to 70.7 | Improved |
| patch size sweep for patch-random (changing number of autoregressive steps without segments) (ablation) | patch-random autoregression · CIFAR10 | linear probing: LIN ranges 59.79 to 75.53 across P; best at P=4 only | No clear effect |
| RandSAC-Blob vs MAE and DINO on low-data CIFAR10/100 | RandSAC low-data pretraining · CIFAR10 and CIFAR100 | linear and fine-tune accuracy: CIFAR10 LIN 93.9 vs DINO 89.0; CIFAR100 LIN 69.7 (Square) vs DINO 65.78 | Improved |
| RandSAC-Square (16->4) pretraining on ImageNet-1K vs MAE | ViT-B ImageNet-1K pretraining · ImageNet-1K | linear and fine-tune accuracy: LIN 68.9 vs MAE 68.0 (higher); FT 83.9 vs MAE 83.6 (higher, but DINO/MoCo linear higher) | Mixed |
Closest papers by meaning
- Understand Before You Generate: Self-Guided Training for Autoregressive Image Generation (NeurIPS 2025 · similarity 0.87)
- Rejuvenating image-GPT as Strong Visual Representation Learners (ICML 2024 · similarity 0.87)
- Beyond Random Augmentations: Pretraining with Hard Views (ICLR 2025 · similarity 0.87)
- Data-efficient Large Vision Models through Sequential Autoregression (ICML 2024 · similarity 0.87)
- Seg-VAR:Image Segmentation with Visual Autoregressive Modeling (NeurIPS 2025 · similarity 0.87)
- Visual Representation Learning with Stochastic Frame Prediction (ICML 2024 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.
