AI Paper Atlas

Research map / Architectures and efficiency / Attention and transformers

Self-supervision through Random Segments with Autoregressive Coding (RandSAC)

Tianyu Hua, Yonglong Tian, Sucheng Ren, Michalis Raptis, Hang Zhao, Leonid Sigal

ICLR 2023 · Architectures and efficiency › Attention and transformers · approach: Token merging and masked pretraining · 5 citations (OpenAlex)

Read the paperCheck your idea against 137,217 papers

AI-extracted. The measured effects below were read by AI from the paper's full text (no summary yet). Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Measured effects reported in the paper

What was changedWhereMeasuredEffect
randomized serialization of patches instead of raster order (ablation)patch tokenization, autoregressive ViT · CIFAR10 and ImageNet100linear probing and fine-tuning accuracy: CIFAR10 LIN +20 points (55.53 to 75.53); IN100 LIN 49.35 to 53.02Improved
patch tokenization (P=4) instead of pixel tokenization (ablation)raster-order autoregressive model · CIFAR10linear probing and fine-tuning accuracy: LIN 41.70 to 55.53; FT 59.35 to 78.67Improved
grouping patches into square segments (M=2) with random serialization (ablation)patch-random autoregression · CIFAR10linear probing and fine-tuning accuracy: LIN 75.53 to 81.38; FT 87.52 to 91.38Improved
grouping patches into blob segments (K=5) instead of single patches (ablation)patch-random autoregression · CIFAR10 and ImageNet100linear and fine-tune accuracy: CIFAR10 LIN 75.53 to 82.52; IN100 LIN 53.02 to 65.00Improved
making segments spatially incoherent (random shuffle of tokens) (ablation)blob segments · CIFAR10linear probing: LIN ~80.9-82.5 coherent vs ~76.7-78.6 shuffledHurt
two-level hierarchical segment serialization (16->4) instead of flat 16 square segments (ablation)square RandSAC · CIFAR10 and ImageNet100linear and fine-tune accuracy: CIFAR10 LIN 81.38 to 82.46; IN100 LIN 64.90 to 65.81Improved
trainable skip connections from encoder layers to decoder layers (ablation)masked transformer decoder for RandSAC · CIFAR10 and ImageNet100linear probing: CIFAR10 LIN 88.0 to 89.5; IN100 LIN 65.8 to 70.7Improved
patch size sweep for patch-random (changing number of autoregressive steps without segments) (ablation)patch-random autoregression · CIFAR10linear probing: LIN ranges 59.79 to 75.53 across P; best at P=4 onlyNo clear effect
RandSAC-Blob vs MAE and DINO on low-data CIFAR10/100RandSAC low-data pretraining · CIFAR10 and CIFAR100linear and fine-tune accuracy: CIFAR10 LIN 93.9 vs DINO 89.0; CIFAR100 LIN 69.7 (Square) vs DINO 65.78Improved
RandSAC-Square (16->4) pretraining on ImageNet-1K vs MAEViT-B ImageNet-1K pretraining · ImageNet-1Klinear and fine-tune accuracy: LIN 68.9 vs MAE 68.0 (higher); FT 83.9 vs MAE 83.6 (higher, but DINO/MoCo linear higher)Mixed

Closest papers by meaning

Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.

Open scime draft

Get the next scime Dispatch and a monthly field update

Free. We send a confirmation link first; every email has a one-click unsubscribe. Privacy