Research map / Architectures and efficiency / Attention and transformers
Towards Understanding Self-Pretraining for Sequence Classification
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Replicates Amos et al.'s self-pretraining (SPT) results on the Long-Range Arena and ablates depth, data source, freezing, and hybrid initialization to argue that label supervision struggles to learn query-key attention patterns, which SPT provides as proximity-biased attention.
Problem
Why does self-pretraining (masked reconstruction on the same task data) substantially improve Transformers on sequence classification such as the Long-Range Arena, and why does standard supervised training from scratch fail?
Approach
Ablations show SPT gains persist in 1-layer models and with swapped pretraining data, and that initializing only W_Q and W_K from SPT recovers the gain; a toy task and a simplified theory show mean-pooled label loss is locally blind to certain attention-score directions that masked reconstruction detects.
Key results
- Reproduction of Amos et al. SPT gains: CIFAR10 from 0.499 to 0.72 and PathFinder from 0.67 to 0.85 with 200 SPT epochs.
- Initializing only W_Q and W_K from SPT in 1-layer models gives CIFAR10 0.607 vs 0.486 from scratch and PathFinder 0.704 vs 0.672.
- Freezing attention at random init leaves from-scratch ListOps, Retrieval and Text accuracy unchanged, while PathFinder drops from 0.672 to 0.50.
Surprising findings
- Freezing attention weights at random initialization does not hurt from-scratch CIFAR10 accuracy and slightly improves it (0.535 vs 0.503).
- SPT gains of about 10% in training and test accuracy appear on a toy task with 1-layer attention, and SPT with a dataset swap still improves validation accuracy on CIFAR10.
Limitations
- Multi-seed ablations focus on CIFAR10 and PathFinder; PathX is excluded and several large runs are single-seed.
- Toy task and theory simplify architecture, data distribution, and attention parametrization.
- The theory isolates one possible obstruction rather than a full explanation.
Experimental setup
Datasets: ListOps CIFAR10 PathFinder Retrieval Text GunPoint GunPointAgeSpan ECG200 ECG5Days synthetic toy trajectory task
Benchmarks: Long-Range Arena (ListOps, CIFAR10, PathFinder, Retrieval, Text) UCR (GunPoint, GunPointAgeSpan, ECG200, ECG5Days)
Metrics: peak held-out validation accuracy training accuracy Frobenius displacement
Baselines: From-scratch training ALiBi FIRE RoPE
Builds on: Amos et al. 2024 self-pretraining Long-Range Arena ALiBi FIRE RoPE
Closest papers by meaning
- Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors (ICLR 2024 · similarity 0.87)
- On the Optimal Memorization Capacity of Transformers (ICLR 2025 · similarity 0.86)
- How Transformers Learn Causal Structure with Gradient Descent (ICML 2024 · similarity 0.86)
- Non-asymptotic Convergence of Training Transformers for Next-token Prediction (NeurIPS 2024 · similarity 0.86)
- How Transformers Learn Structured Data: Insights From Hierarchical Filtering (ICML 2025 · similarity 0.86)
- No Train, all Gain: Self-Supervised Gradients Improve Deep Frozen Representations (NeurIPS 2024 · similarity 0.85)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.