Research map / Architectures and efficiency / Attention and transformers
Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Shows that self-pretraining (denoising pretraining on only the downstream task data) greatly improves Transformers and S4 on Long Range Arena, closing the gap between architectures and making diagonal linear RNNs competitive.
Problem
Long-sequence benchmarks such as Long Range Arena are evaluated by training models from random initialization, which may underestimate architectures (notably Transformers) and yield unfair comparisons relative to pretrained practice.
Approach
Pretrain each architecture with a standard denoising objective (causal or masked sequence modeling) solely on the downstream task's training data before finetuning, which acts as a data-driven prior and enables fair architecture comparison without extra data.
Key results
- Self-pretrained Transformers with causal SPT reach 81.81 average on LRA (6 tasks) versus 53.66 for the randomly initialized Transformer baseline from prior work.
- With SPT, S4 reaches 87.11% on PathX-256, improving the prior best of 63.1% (67 to 87 reported in the intro) by about 20 absolute points.
- Diagonal linear RNN (DLR) with SPT achieves 83.4 average versus 82.8 for trained-from-scratch S4, close to SPT S4 at 84.5.
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| self-pretrain Transformer with masked sequence modeling | Transformer · LRA | mean accuracy: 53.66 -> 81.34 | Helped |
| self-pretrain Transformer with causal sequence modeling | Transformer · LRA | mean accuracy: 53.66 -> 81.81 | Helped |
| use masked rather than causal SPT (ablation) | Transformer · LRA | mean accuracy: 81.34 vs 81.81 | No clear effect |
| add masked SPT and rotary embeddings | Transformer · LRA Avg. (without PathX-256) | accuracy: 74.28 -> 86.21 | Helped |
| add masked SPT | S4 · PathX-256 | accuracy: 67.82 -> 87.11 | Helped |
| add masked SPT | S4 · LRA (ListOps, Text, Image, PathX) | mean accuracy: 82.85 -> 84.47 | Helped |
| use DLR instead of S4, with SPT (ablation) | SSM · LRA (ListOps, Text, Image, PathX) | mean accuracy: 84.47 -> 83.37 | Hurt |
| replace structured initialization with random initialization before SPT (ablation) | S4 · LRA (ListOps, Text, Image, PathX) | mean accuracy: 82.85 -> 82.75 | No clear effect |
| add SPT | Transformer · Speech Commands (bidirectional) | accuracy: 86.93 -> 91.49 | Helped |
| add SPT | Transformer · sCIFAR | accuracy: 79.41 -> 90.29 | Helped |
| add SPT | S4 · BIDMC HR | R2: 0.999 -> 0.999 | No clear effect |
| allocate 20% of a fixed training budget to SPT | Transformer · Image | accuracy: 75.04 -> 84.45 | Helped |
Surprising findings
- Vanilla Transformers with self-pretraining match S4 on Long Range Arena (86.2 vs 86.1 average), contrary to prior reports of poor Transformer performance.
- Structured S4 parameterization and initialization become largely redundant under self-pretraining, as a plain diagonal linear RNN nearly matches S4.
Limitations
- Gains from SPT are largest on small or medium tasks and some tasks show modest gains for S4 (BIDMC near ceiling).
- Some Transformer configurations differ from prior LRA setups (rotary embeddings, block-sparse attention for 16K+ sequences), and PathX-256 uses max pooling, so comparisons are not fully identical to original setups.
- Compute-tied experiments show SPT with limited budget still lags the unrestricted setting, implying the evaluation scheme is costly.
Experimental setup
Datasets: ListOps IMDb (character-level Text) AAN (Retrieval) CIFAR-10 grayscale (Image) Pathfinder PathX PathX-256 Speech Commands sCIFAR BIDMC The Pile
Benchmarks: Long Range Arena PathX-256 Speech Commands sCIFAR BIDMC
Metrics: accuracy average accuracy R2
Baselines: S4 SPADE MEGA Transformers + Rotary Pythia 70M DLR Linformer BigBird Performer
Builds on: S4 HiPPO Long Range Arena diagonal linear RNN self-supervised pretraining on downstream data
Closest papers by meaning
- Reinforced Fast Weights via Next-Sequence Prediction (NeurIPS 2026 · similarity 0.87)
- Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing (ICLR 2025 · similarity 0.87)
- HOPE for a Robust Parameterization of Long-memory State Space Models (ICLR 2025 · similarity 0.87)
- Local to Global: Learning Dynamics and Effect of Initialization for Transformers (NeurIPS 2024 · similarity 0.87)
- Towards Understanding Self-Pretraining for Sequence Classification (NeurIPS 2026 · similarity 0.87)
- Benchmarking Optimizers for Large Language Model Pretraining (NeurIPS 2026 · similarity 0.86)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.