Research map / Label-efficient and robust learning / Self-supervised and contrastive

PEIRA: Learning Predictive Encoders through Inter-View Regressor Alignment

NeurIPS 2026 · Label-efficient and robust learning › Self-supervised and contrastive · approach: Data-efficient and robust self-supervision · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Analyzes JEPA-style self-distillation with a linear regressor via nonlinear CCA, then proposes PEIRA, an unconstrained objective (negative trace of the optimal ridge predictor) whose stable equilibria are global minimizers spanning top canonical subspaces, and gives a stochastic EMA-based training method evaluated on ImageNet-1K and CIFAR-10.

Problem

Self-distillation methods (SimSiam, BYOL, DINO, I-JEPA) work well empirically in non-contrastive SSL but lack a well-defined objective and their learned representations and collapse behavior are poorly understood.

Approach

Derive the equilibria and stability of JEPA-style self-distillation with an optimal ridge regressor as nonlinear CCA subspaces, then define PEIRA's objective as the negative trace of the optimal regularized predictor plus a norm penalty, whose only stable equilibria are nontrivial global minimizers; train it via an auxiliary objective with EMA estimates of the regressor, avoiding differentiation t

Key results

• On ImageNet-1K (ResNet-50, 100 epochs) PEIRA reaches 66.50 ± 0.02% linear-probe top-1, matching SIGReg (66.26%) and trailing VICReg (68.81%) by about 2.3 points.
• On CIFAR-10 (ResNet-18, 1000 epochs) PEIRA reaches 90.97 ± 0.10%, matching VICReg (90.92%) and trailing SIGReg (92.34%) by about 1.4 points.
• Top-1 accuracy varies by only 1.36% across lambda in [0.025, 0.5].

Surprising findings

• Self-distillation dynamics with a linear predictor admit a stable collapsed equilibrium, while PEIRA's trivial solution is unstable for rank r_max >= 2.
• Negative PEIRA objective correlates strongly with downstream linear-probe accuracy, enabling label-free model selection.

Limitations

• Theory assumes an optimal (fast) predictor and distinct canonical correlations; discrete stochastic updates are not analyzed.
• Experiments are described as preliminary; PEIRA trails VICReg on ImageNet-1K and SIGReg on CIFAR-10.
• Parametric encoder training departs from the infinite-dimensional theory; code to be released upon publication.

Experimental setup

Datasets: ImageNet-1K CIFAR-10

Benchmarks: ImageNet-1K CIFAR-10

Metrics: linear-probe top-1 accuracy

Baselines: VICReg SIGReg (LeJEPA) BYOL SimSiam SimCLR

Builds on: JEPA VICReg SimSiam BYOL stochastic compositional optimization

Closest papers by meaning

• Joint‑Embedding vs Reconstruction: Provable Benefits of Latent Space Prediction for Self‑Supervised Learning (NeurIPS 2025 · similarity 0.86)
• Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures (NeurIPS 2026 · similarity 0.86)
• Collapse-Proof Non-Contrastive Self-Supervised Learning (ICML 2025 · similarity 0.86)
• Harnessing small projectors and multiple views for efficient vision pretraining (NeurIPS 2024 · similarity 0.85)
• seq-JEPA: Autoregressive Predictive Learning of Invariant-Equivariant World Models (NeurIPS 2025 · similarity 0.85)
• Learning from Memory: Non-Parametric Memory Augmented Self-Supervised Learning of Visual Features (ICML 2024 · similarity 0.85)