Research map / Architectures and efficiency / RNNs and normalization
Efficiency for Free: Ideal Data Are Transportable Representations
Read the paperCheck your idea against 108,967 papers
AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.
Proposes ReLA, an optimization-free method that uses a freely available prior model to relabel data with transportable representation targets, plus a loss term that accelerates (self-)supervised learning, cutting BYOL cost by 50% on ImageNet-1K.
Problem
Representation learning is costly in data and compute; self-supervised learning is inefficient relative to supervised learning due to self-generated targets, and dataset distillation is itself computationally expensive. The paper asks how task- and architecture-agnostic publicly available pre-traine
Approach
Well-trained prior models converge to linearly transportable representations, so their representations of unlabeled samples serve as efficient targets; combining weak augmentation for samples with an adaptively decaying auxiliary loss on these prior targets accelerates any SSL method at no extra per-step cost once the prior term is switched off.
Key results
- On ImageNet-1K with ResNet-50, BYOL with ReLA using a CIFAR-10-pretrained ResNet-18 prior reaches the accuracy of full-budget BYOL at 50% of the cost, as stated in the abstract.
- On CIFAR-100, BYOL-ReLA with 10% of the budget (e.g., with IN1K-T or CLIP-RN50 prior) exceeds BYOL trained with the full budget's 41.4% at 50% budget; ReLA-BYOL at 10% reaches 56.4% (IN1K-T) vs 41.4% for BYOL at 50%.
- Across seven SSL methods on CIFAR-10 at 10% budget, ReLA with a random-init prior improves over original training for most methods (e.g., SimSiam 45.8 to 66.7, Barlow 63.7 to 68.8).
Measured effects reported in the paper
| What was changed | Where | Measured | Effect |
|---|---|---|---|
| Add ReLA with a CLIP-RN50 prior to BYOL | BYOL on CIFAR-10, 10% budget · CIFAR-10 | linear-probe accuracy: 58.3 ± 0.1 -> 82.0 ± 0.1 | Helped |
| Add ReLA with an IN1K-T prior to BYOL | BYOL on CIFAR-100, 10% budget · CIFAR-100 | linear-probe accuracy: 26.9 ± 0.2 -> 56.4 ± 0.2 | Helped |
| Add ReLA with an IN1K-T prior to BYOL | BYOL on Tiny-ImageNet, 10% budget · Tiny-ImageNet | linear-probe accuracy: 25.1 ± 0.3 -> 41.6 ± 0.1 | Helped |
| Add ReLA with a CLIP-RN50 prior to BYOL | BYOL on ImageNet-1K, 10% budget · ImageNet-1K | linear-probe accuracy: 44.5 ± 0.1 -> 56.2 ± 0.1 | Helped |
| Use ReLA with an IN1K-T prior at 20% budget rather than full-budget BYOL | BYOL on CIFAR-100 · CIFAR-100 | linear-probe accuracy: 52.5 ± 0.3 -> 59.5 ± 0.1 | Helped |
| Use ReLA with an IN1K-T prior at 20% budget rather than full-budget BYOL | BYOL on Tiny-ImageNet · Tiny-ImageNet | linear-probe accuracy: 43.6 ± 0.3 -> 44.1 ± 0.1 | Helped |
| Add ReLA with a CLIP-RN50 prior to SimCLR | SimCLR on CIFAR-10, 10% budget · CIFAR-10 | linear-probe accuracy: 70.7 ± 0.2 -> 76.4 ± 0.1 | Helped |
| Add ReLA with a CLIP-RN50 prior to Vicreg | Vicreg on Tiny-ImageNet, 10% budget · Tiny-ImageNet | linear-probe accuracy: 34.0 ± 0.1 -> 39.1 ± 0.2 | Helped |
| Use a stronger CLIP-RN50 prior rather than a randomly initialized prior with ReLA | BYOL ReLA on CIFAR-10, 10% budget · CIFAR-10 | linear-probe accuracy: 71.4 ± 0.0 -> 82.0 ± 0.1 | Helped |
| Use a stronger CLIP-RN50 prior rather than a randomly initialized prior with ReLA | BYOL ReLA on Tiny-ImageNet, 10% budget · Tiny-ImageNet | linear-probe accuracy: 34.5 ± 0.3 -> 39.6 ± 0.4 | Helped |
| Use adaptive rather than static ReLA weight scheduling (ablation) | BYOL ReLA across prior-model settings · Multiple settings | linear-probe accuracy | Helped |
| Use larger ReLA weights with a strong prior and smaller ReLA weights with a weak prior (ablation) | BYOL ReLA · Multiple settings | linear-probe accuracy | Mixed |
Surprising findings
- A randomly initialized ResNet-18 used as the prior model still accelerates learning, which the authors frame as weak-to-strong supervision with pseudo targets.
- Using CIFAR-10-pretrained ResNet-18 as the prior for ImageNet-1K training is competitive with CLIP-RN50 as prior.
Limitations
- Theoretical analysis is restricted to a simplified bimodal Gaussian mixture and a single-hidden-layer network.
- Prior models downloaded online are not consistently robust, so the method depends on prior quality and needs an adaptive lambda schedule.
- Reported computational savings are measured in training steps/time rather than a full cost accounting that includes the one-time target generation.
Experimental setup
Datasets: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Benchmarks: CIFAR-10 CIFAR-100 Tiny-ImageNet ImageNet-1K
Metrics: linear probing accuracy computational budget (training steps fraction)
Baselines: BYOL SimCLR Barlow Twins DINO MoCo SimSiam SwAV VICReg SRe2L RDED
Builds on: BYOL SRe2L RDED UniGrad
Closest papers by meaning
- DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing (AAAI 2024 · similarity 0.85)
- Optimal Representation Size: High-Dimensional Analysis of Pretraining and Linear Probing (NeurIPS 2026 · similarity 0.85)
- Provable Sample-Efficient Transfer Learning Conditional Diffusion Models via Representation Learning (NeurIPS 2025 · similarity 0.83)
- When Dynamic Data Selection Meets Data Augmentation: Achieving Enhanced Training Acceleration (ICML 2025 · similarity 0.83)
- You Don’t Need Domain-Specific Data Augmentations When Scaling Self-Supervised Learning (NeurIPS 2024 · similarity 0.83)
- Towards Scalable and Versatile Weight Space Learning (ICML 2024 · similarity 0.83)
Writing a paper in this area? scime draft is a free LaTeX editor with AI included, real-time co-authors and a check of your references and venue rules before you submit. It also imports OpenAI Prism projects.