Research map / Architectures and efficiency / RNNs and normalization

Drive vs. Decay: On the Training Dynamics of Joint-Embedding Predictive Architectures

NeurIPS 2026 · Architectures and efficiency › RNNs and normalization · approach: Layer-wise analysis of language models · 0 citations (OpenAlex)

Read the paperCheck your idea against 108,967 papers

AI-extracted summary. Written by AI from the paper's full text. Numbers and claims can be wrong or missing context, so check the paper before you cite or build on it.

Develops a linear drive-decay stability theory for JEPA training dynamics (per-mode ratio mu = gamma/sigma), validates it on ~800 Tabular-JEPA configs, and proposes ResidualPred, an identity-biased transformer predictor that improves tabular and I-JEPA results.

Problem

JEPAs are prone to representation collapse, and existing anti-collapse fixes are empirical heuristics lacking an explicit training-dynamics account of the original JEPA objective under realistic update rules.

Approach

Linearise the coupled JEPA gradient flow (context encoder, EMA target, predictor) around the trivial fixed point; under approximate spectral decoupling the escape criterion reduces to a per-mode ratio mu = gamma/sigma between a driving operator (view alignment) and a decay operator (predictor and data covariance). Predictor scale, masking ratio, EMA and other heuristics are then viewed as shifts o

Key results

• Across 810 runs, the count of super-critical modes tracks effective rank with rho = +0.91 on helena.
• ResidualPred improves CIFAR-100 I-JEPA linear probe from 36.83% to 40.85% and T-JEPA helena accuracy from 8.6% to 21.3%.
• On an ImageNet-1k pilot at matched budget, I-JEPA+ResidualPred+SIGReg reaches 32.23% linear accuracy vs 25.22% for I-JEPA.

Surprising findings

• Final training loss is a poor diagnostic: near-zero MSE can accompany degenerate embeddings.
• Effective rank on images is necessary for accuracy without being sufficient.

Limitations

• Theory is derived in a linear setting and relies on an approximate spectral-decoupling assumption that is violated on some datasets (e.g. jannis).
• The extension to images is empirical; ImageNet-1k results are matched-budget pilots, not the full official I-JEPA recipe.
• ResidualPred extends identity-bias to attention by analogy rather than formal derivation.

Experimental setup

Datasets: aloi helena jannis CIFAR-10 CIFAR-100 STL-10 ImageNet-100 ImageNet-1k

Benchmarks: aloi helena jannis CIFAR-10 CIFAR-100 STL-10 ImageNet-100 ImageNet-1k

Metrics: linear probe accuracy effective rank test accuracy spearman correlation

Baselines: T-JEPA I-JEPA SIGReg mimetic initialization

Builds on: I-JEPA T-JEPA SIGReg VICReg mimetic initialization

Closest papers by meaning

• LeWorldModel: Stable End-to-End Joint-Embedding Predictive Architecture from Pixels (NeurIPS 2026 · similarity 0.89)
• Denoising with a Joint-Embedding Predictive Architecture (ICLR 2025 · similarity 0.87)
• How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networks (NeurIPS 2024 · similarity 0.87)
• Scaling Collapse Reveals Universal Dynamics in Compute-Optimally Trained Neural Networks (ICML 2025 · similarity 0.87)
• Connecting Joint-Embedding Predictive Architecture with Contrastive Self-supervised Learning (NeurIPS 2024 · similarity 0.86)
• Dynamics of the Transformer Residual Stream: Coupling Spectral Geometry to Network Topology (NeurIPS 2026 · similarity 0.86)